Nov 03 15:05:23.189549 ip-10-0-139-207 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 15:05:23.189560 ip-10-0-139-207 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 15:05:23.189570 ip-10-0-139-207 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 15:05:23.189894 ip-10-0-139-207 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 15:05:33.314650 ip-10-0-139-207 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 15:05:33.314664 ip-10-0-139-207 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 28580130b2ac485bb89bc53f0282f264 -- Nov 03 15:07:28.306934 ip-10-0-139-207 systemd[1]: Starting Kubernetes Kubelet... Nov 03 15:07:28.623695 ip-10-0-139-207 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 15:07:28.623695 ip-10-0-139-207 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 15:07:28.623695 ip-10-0-139-207 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 15:07:28.623695 ip-10-0-139-207 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 15:07:28.627448 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627372 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:07:28.627448 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627447 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:07:28.627448 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627451 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627455 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627458 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627460 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627463 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627466 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627468 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627471 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627474 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627477 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627479 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627482 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627485 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627487 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627490 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627492 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627495 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627497 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627500 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:07:28.627532 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627510 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627513 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627515 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627518 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627521 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627523 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627526 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627528 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627531 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627533 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627536 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627539 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627542 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627544 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627547 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627550 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627552 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627555 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627557 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627559 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:07:28.627970 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627562 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627564 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627566 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627569 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627571 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627573 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627576 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627579 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627582 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627584 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627587 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627589 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627591 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627594 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627597 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627600 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627602 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627605 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627608 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:07:28.628425 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627612 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627616 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627619 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627621 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627624 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627627 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627629 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627631 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627634 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627637 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627639 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627642 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627644 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627647 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627661 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627664 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627667 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627670 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627672 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627675 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:07:28.628877 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627677 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627680 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627684 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627688 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627691 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627693 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627697 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627700 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627703 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627706 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627709 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627711 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627714 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627716 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627719 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627721 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627724 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627726 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627729 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627731 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:07:28.629387 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627733 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627736 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627738 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627741 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627744 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.627747 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628485 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628492 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628496 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628498 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628501 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628504 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628506 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628509 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628512 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628515 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628517 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628520 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628522 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628525 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:07:28.629915 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628528 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628531 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628533 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628536 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628539 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628541 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628545 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628550 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628553 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628555 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628558 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628561 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628563 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628566 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628568 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628570 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628573 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628576 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628579 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:07:28.630421 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628581 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628584 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628586 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628589 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628591 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628594 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628596 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628598 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628602 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628604 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628607 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628609 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628612 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628615 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628617 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628619 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628622 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628624 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628627 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628629 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:07:28.630865 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628631 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628634 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628636 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628638 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628641 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628643 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628646 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628648 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628665 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628669 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628674 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628677 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628681 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628685 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628688 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628692 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628696 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628700 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628704 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:07:28.631323 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628707 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628712 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628716 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628720 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628724 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628728 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628733 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628737 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628740 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628743 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628745 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628748 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628750 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628753 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628755 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628758 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628760 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628763 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628765 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628768 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:07:28.631796 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628770 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628772 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628776 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628778 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628782 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628787 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628790 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628793 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628796 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628799 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628802 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628805 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628807 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.628810 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629940 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629948 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629952 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629956 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629959 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:07:28.632256 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629962 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629965 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629968 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629970 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629973 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629976 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629979 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629982 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629985 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629988 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629990 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629993 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629995 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.629998 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630001 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630004 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630006 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630009 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630012 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630015 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:07:28.632697 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630017 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630019 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630022 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630024 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630027 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630030 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630032 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630035 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630038 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630041 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630044 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630047 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630049 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630052 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630054 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630057 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630060 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630063 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630066 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630068 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:07:28.633189 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630071 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630073 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630076 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630078 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630081 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630083 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630098 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630101 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630103 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630106 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630108 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630110 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630113 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630115 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630119 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630123 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630126 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630129 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630132 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:07:28.633670 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630135 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630137 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630140 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630143 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630146 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630148 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630150 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630153 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630155 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630158 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630161 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630163 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630166 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630168 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630171 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630173 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630176 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630178 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630181 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630183 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:07:28.634216 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630186 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630188 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630190 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630193 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630195 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630198 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630200 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630203 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630205 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630208 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630212 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630214 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630217 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630219 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630222 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630225 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630228 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630230 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630233 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630236 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:07:28.634691 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630238 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.630241 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.630983 2573 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.630992 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.630998 2573 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631002 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631006 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631009 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631013 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631017 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631021 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631024 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631027 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631030 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631038 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631041 2573 flags.go:64] FLAG: --cgroup-root="" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631044 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631047 2573 flags.go:64] FLAG: --client-ca-file="" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631050 2573 flags.go:64] FLAG: --cloud-provider="external" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631053 2573 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631057 2573 flags.go:64] FLAG: --cluster-domain="" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631060 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631063 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631066 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 15:07:28.635149 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631069 2573 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631074 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631076 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631079 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631083 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631086 2573 flags.go:64] FLAG: --contention-profiling="false" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631089 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631092 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631095 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631098 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631102 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631105 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631108 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631110 2573 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631113 2573 flags.go:64] FLAG: --enable-server="true" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631116 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631120 2573 flags.go:64] FLAG: --event-burst="100" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631123 2573 flags.go:64] FLAG: --event-qps="50" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631126 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631129 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631132 2573 flags.go:64] FLAG: --eviction-hard="" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631136 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631139 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631142 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631145 2573 flags.go:64] FLAG: --eviction-soft="" Nov 03 15:07:28.635707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631147 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631150 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631154 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631157 2573 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631160 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631163 2573 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631165 2573 flags.go:64] FLAG: --feature-gates="" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631170 2573 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631173 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631176 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631179 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631182 2573 flags.go:64] FLAG: --healthz-port="10248" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631185 2573 flags.go:64] FLAG: --help="false" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631188 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631191 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631194 2573 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631197 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631200 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631203 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631206 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631209 2573 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631212 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631214 2573 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 15:07:28.636338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631217 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631220 2573 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631223 2573 flags.go:64] FLAG: --kube-reserved="" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631226 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631228 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631231 2573 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631234 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631238 2573 flags.go:64] FLAG: --lock-file="" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631240 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631243 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631246 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631254 2573 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631257 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631260 2573 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631263 2573 flags.go:64] FLAG: --logging-format="text" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631266 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631269 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631272 2573 flags.go:64] FLAG: --manifest-url="" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631275 2573 flags.go:64] FLAG: --manifest-url-header="" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631279 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631282 2573 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631286 2573 flags.go:64] FLAG: --max-pods="110" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631290 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631293 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631295 2573 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 15:07:28.636888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631298 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631301 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631304 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631307 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631314 2573 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631317 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631320 2573 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631323 2573 flags.go:64] FLAG: --pod-cidr="" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631326 2573 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631329 2573 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631332 2573 flags.go:64] FLAG: --pods-per-core="0" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631335 2573 flags.go:64] FLAG: --port="10250" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631338 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631340 2573 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0af2ebdf5a7bca374" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631343 2573 flags.go:64] FLAG: --qos-reserved="" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631346 2573 flags.go:64] FLAG: --read-only-port="10255" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631349 2573 flags.go:64] FLAG: --register-node="true" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631352 2573 flags.go:64] FLAG: --register-with-taints="" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631357 2573 flags.go:64] FLAG: --registry-burst="10" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631360 2573 flags.go:64] FLAG: --registry-qps="5" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631363 2573 flags.go:64] FLAG: --reserved-cpus="" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631365 2573 flags.go:64] FLAG: --reserved-memory="" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631369 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631372 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631375 2573 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631378 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 15:07:28.637543 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631381 2573 flags.go:64] FLAG: --runonce="false" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631384 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631387 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631390 2573 flags.go:64] FLAG: --seccomp-default="false" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631393 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631396 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631399 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631402 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631405 2573 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631407 2573 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631410 2573 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631413 2573 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631416 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631419 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631421 2573 flags.go:64] FLAG: --system-cgroups="" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631424 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631429 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631432 2573 flags.go:64] FLAG: --tls-cert-file="" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631434 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631439 2573 flags.go:64] FLAG: --tls-min-version="" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631441 2573 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631444 2573 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631447 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631450 2573 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631454 2573 flags.go:64] FLAG: --v="2" Nov 03 15:07:28.638177 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631459 2573 flags.go:64] FLAG: --version="false" Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631463 2573 flags.go:64] FLAG: --vmodule="" Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631466 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631469 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631597 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631603 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631605 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631608 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631611 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631613 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631616 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631618 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631621 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631626 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631629 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631632 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631635 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631637 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631640 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631642 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:07:28.638756 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631645 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631647 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631661 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631665 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631667 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631670 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631672 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631675 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631677 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631681 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631683 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631688 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631690 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631693 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631695 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631698 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631700 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631704 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631707 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:07:28.639214 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631709 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631712 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631714 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631717 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631720 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631722 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631725 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631728 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631730 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631733 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631735 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631738 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631740 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631743 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631745 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631748 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631750 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631753 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631755 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631757 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:07:28.639662 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631760 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631762 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631765 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631767 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631771 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631774 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631776 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631779 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631781 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631784 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631787 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631790 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631792 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631795 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631797 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631800 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631803 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631805 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631808 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:07:28.640123 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631811 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631814 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631816 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631819 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631823 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631827 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631829 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631832 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631834 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631837 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631839 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631842 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631844 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631847 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631849 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631852 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631854 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631858 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631860 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631863 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:07:28.640605 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631866 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:07:28.641118 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631868 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:07:28.641118 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631871 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:07:28.641118 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631875 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:07:28.641118 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631877 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:07:28.641118 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631880 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:07:28.641118 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631882 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:07:28.641118 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631885 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:07:28.641118 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631888 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:07:28.641118 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631890 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:07:28.641118 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631893 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:07:28.641118 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631895 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:07:28.641118 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.631945 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 15:07:28.641118 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631982 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:07:28.641118 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631987 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:07:28.641118 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631990 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631992 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631996 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.631998 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632001 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632004 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632007 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632009 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632012 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632015 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632017 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632020 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632022 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632025 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632027 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632032 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632036 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632039 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632042 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:07:28.641472 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632044 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632049 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632051 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632054 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632056 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632059 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632061 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632063 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632066 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632068 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632071 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632074 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632077 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632079 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632082 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632084 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632087 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632089 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632091 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632094 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:07:28.641930 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632097 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632100 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632102 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632105 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632107 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632109 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632112 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632114 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632118 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632120 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632123 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632125 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632128 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632134 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632137 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632140 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632143 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632146 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632148 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632151 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:07:28.642423 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632153 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632156 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632158 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632161 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632163 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632166 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632169 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632171 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632174 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632176 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632179 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632181 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632184 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632186 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632188 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632191 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632193 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632196 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632198 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:07:28.642886 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632201 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632204 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632207 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632209 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632212 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632214 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632218 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632221 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632223 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632226 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632229 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632231 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632234 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632236 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632238 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632241 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632244 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632247 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632249 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632252 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:07:28.643424 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632255 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632257 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632260 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632263 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632266 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.632269 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.638698 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.638714 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.638729 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.638733 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638785 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638791 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638795 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638798 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638801 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638804 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638807 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638810 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638813 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638815 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:07:28.644030 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638818 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638821 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638824 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638826 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638829 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638831 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638834 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638837 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638839 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638842 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638844 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638847 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638849 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638852 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638854 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638857 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638859 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638862 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638864 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:07:28.644479 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638868 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638871 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638874 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638876 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638879 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638881 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638884 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638886 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638889 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638891 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638894 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638896 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638899 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638901 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638904 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638907 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638912 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638914 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638917 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:07:28.644974 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638920 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638922 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638925 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638927 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638930 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638932 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638935 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638937 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638939 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638942 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638944 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638947 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638949 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638952 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638955 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638958 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638960 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638963 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638966 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638968 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:07:28.645496 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638971 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638973 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638976 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638978 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638980 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638983 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638985 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638987 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638990 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638992 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638995 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.638997 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639000 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639002 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639004 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639007 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639010 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639012 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639015 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:07:28.646254 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639017 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:07:28.646900 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639020 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:07:28.646900 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639022 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:07:28.646900 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639024 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:07:28.646900 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639027 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:07:28.646900 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639029 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:07:28.646900 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639032 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:07:28.646900 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639035 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:07:28.646900 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639038 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:07:28.646900 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639042 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:07:28.646900 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639045 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:07:28.646900 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639048 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:07:28.646900 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639051 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:07:28.646900 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639054 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:07:28.646900 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639056 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:07:28.646900 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639058 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:07:28.646900 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639061 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:07:28.646900 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639063 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:07:28.646900 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639066 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:07:28.647372 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.639091 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 15:07:28.647372 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639223 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:07:28.647372 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639229 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:07:28.647372 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639232 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:07:28.647372 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639235 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:07:28.647372 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639238 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:07:28.647372 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639240 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:07:28.647372 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639242 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:07:28.647372 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639245 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:07:28.647372 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639249 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:07:28.647372 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639252 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:07:28.647372 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639255 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:07:28.647372 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639258 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:07:28.647372 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639261 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639263 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639266 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639268 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639271 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639273 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639276 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639279 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639281 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639284 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639287 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639289 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639292 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639294 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639297 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639299 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639301 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639304 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639306 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:07:28.647718 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639308 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639311 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639313 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639316 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639318 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639320 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639323 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639325 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639328 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639330 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639333 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639335 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639337 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639340 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639342 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639344 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639347 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639350 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639352 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639355 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:07:28.648282 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639357 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639359 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639362 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639364 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639367 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639369 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639372 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639374 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639377 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639379 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639382 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639384 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639386 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639389 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639391 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639394 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639396 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639398 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639401 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639403 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:07:28.648888 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639406 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639408 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639410 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639413 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639415 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639418 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639420 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639423 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639425 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639428 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639430 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639433 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639435 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639438 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639440 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639443 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639445 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639448 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639450 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639453 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:07:28.649683 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639455 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:07:28.650381 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639458 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:07:28.650381 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639460 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:07:28.650381 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639462 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:07:28.650381 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639465 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:07:28.650381 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639467 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:07:28.650381 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639470 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:07:28.650381 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639472 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:07:28.650381 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639475 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:07:28.650381 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639477 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:07:28.650381 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639480 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:07:28.650381 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639482 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:07:28.650381 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639484 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:07:28.650381 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639487 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:07:28.650381 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639489 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:07:28.650381 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.639511 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 15:07:28.650381 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639542 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:07:28.650786 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639546 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:07:28.650786 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639549 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:07:28.650786 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639552 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:07:28.650786 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639554 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:07:28.650786 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639557 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:07:28.650786 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639559 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:07:28.650786 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639562 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:07:28.650786 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639564 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:07:28.650786 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639568 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:07:28.650786 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639572 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:07:28.650786 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639575 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:07:28.650786 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639577 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:07:28.650786 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639580 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:07:28.650786 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639583 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:07:28.650786 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639585 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:07:28.650786 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639588 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:07:28.650786 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639592 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:07:28.650786 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639596 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639599 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639602 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639604 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639607 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639609 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639612 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639614 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639617 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639619 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639622 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639624 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639626 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639629 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639631 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639634 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639636 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639639 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639641 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:07:28.651196 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639644 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639646 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639649 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639667 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639671 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639674 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639676 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639679 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639681 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639684 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639687 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639690 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639692 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639695 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639697 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639700 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639702 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639704 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639707 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639709 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:07:28.651616 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639712 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639715 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639717 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639719 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639722 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639724 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639727 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639729 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639731 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639734 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639737 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639739 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639742 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639744 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639746 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639749 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639751 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639754 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639756 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639758 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:07:28.652085 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639761 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639766 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639769 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639771 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639774 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639776 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639779 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639781 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639784 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639786 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639788 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639791 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639793 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639796 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639798 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639801 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639804 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639806 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639810 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639812 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:07:28.652823 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639814 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:07:28.653315 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639817 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:07:28.653315 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639819 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:07:28.653315 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639822 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:07:28.653315 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639824 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:07:28.653315 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639827 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:07:28.653315 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639829 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:07:28.653315 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639832 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:07:28.653315 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:28.639834 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:07:28.653315 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.639880 2573 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 15:07:28.653315 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.642026 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 15:07:28.653315 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.642919 2573 server.go:1014] "Starting client certificate rotation" Nov 03 15:07:28.653315 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.643029 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 15:07:28.653315 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.643072 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 15:07:28.653315 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.644631 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 15:07:28.653315 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.646920 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 15:07:28.658259 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.658242 2573 log.go:25] "Validated CRI v1 runtime API" Nov 03 15:07:28.661534 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.661519 2573 log.go:25] "Validated CRI v1 image API" Nov 03 15:07:28.662335 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.662308 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 15:07:28.664213 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.664191 2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 82aa9dbe-8509-4886-b478-f6689de7328d:/dev/nvme0n1p4 ef47aad3-8d74-4d87-ae3e-79887c57d394:/dev/nvme0n1p3] Nov 03 15:07:28.664268 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.664213 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 15:07:28.670322 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.670219 2573 manager.go:217] Machine: {Timestamp:2026-11-03 15:07:28.668954058 +0000 UTC m=+0.268423579 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3210282 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22caec4fb4a97bde5d9d91f7504d99 SystemUUID:ec22caec-4fb4-a97b-de5d-9d91f7504d99 BootID:28580130-b2ac-485b-b89b-c53f0282f264 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6d:15:c3:75:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6d:15:c3:75:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:4e:06:49:65:54 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 15:07:28.670825 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.670816 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 15:07:28.670900 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.670888 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 15:07:28.673237 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.673212 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 15:07:28.673357 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.673238 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-207.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 15:07:28.673951 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.673941 2573 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 15:07:28.673982 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.673954 2573 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 15:07:28.673982 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.673967 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 15:07:28.673982 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.673976 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 15:07:28.675004 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.674995 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 15:07:28.684074 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.684062 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 15:07:28.684199 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.684190 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 15:07:28.684809 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.684801 2573 kubelet.go:493] "Attempting to sync node with API server" Nov 03 15:07:28.684838 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.684813 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 15:07:28.684838 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.684823 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 15:07:28.684838 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.684832 2573 kubelet.go:405] "Adding apiserver pod source" Nov 03 15:07:28.684949 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.684848 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 15:07:28.685159 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.685147 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 15:07:28.687026 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.686994 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 15:07:28.693023 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.693010 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 15:07:28.693698 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.693686 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 15:07:28.693742 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.693717 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 15:07:28.694837 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.694827 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 15:07:28.694873 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.694844 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 15:07:28.694873 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.694850 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 15:07:28.694873 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.694856 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 15:07:28.694873 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.694862 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 15:07:28.694873 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.694868 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 15:07:28.695369 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.695361 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 15:07:28.695397 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.695370 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 15:07:28.695397 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.695377 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 15:07:28.695397 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.695382 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 15:07:28.695397 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.695394 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 15:07:28.695570 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.695404 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 15:07:28.695570 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.695424 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 15:07:28.695570 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.695432 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 15:07:28.698949 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.698936 2573 server.go:1263] "Started kubelet" Nov 03 15:07:28.699053 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.699012 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 15:07:28.699119 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.699021 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 15:07:28.699119 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.699098 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 15:07:28.699669 ip-10-0-139-207 systemd[1]: Started Kubernetes Kubelet. Nov 03 15:07:28.700246 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.700230 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 15:07:28.701381 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.701363 2573 server.go:317] "Adding debug handlers to kubelet server" Nov 03 15:07:28.705586 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:28.705567 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 15:07:28.708582 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.708564 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 15:07:28.709038 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.709020 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 15:07:28.709502 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:28.709133 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-207.us-west-1.compute.internal\" not found" Nov 03 15:07:28.709693 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.709678 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 15:07:28.709764 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.709696 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 15:07:28.709764 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.709750 2573 factory.go:55] Registering systemd factory Nov 03 15:07:28.709764 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.709765 2573 factory.go:223] Registration of the systemd container factory successfully Nov 03 15:07:28.709923 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.709790 2573 reconstruct.go:97] "Volume reconstruction finished" Nov 03 15:07:28.709923 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.709792 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 15:07:28.709923 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.709797 2573 reconciler.go:29] "Reconciler: start to sync state" Nov 03 15:07:28.710051 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.709970 2573 factory.go:153] Registering CRI-O factory Nov 03 15:07:28.710051 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.709981 2573 factory.go:223] Registration of the crio container factory successfully Nov 03 15:07:28.710051 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.710029 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 15:07:28.710051 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.710044 2573 factory.go:103] Registering Raw factory Nov 03 15:07:28.710229 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.710055 2573 manager.go:1199] Started watching for new ooms in manager Nov 03 15:07:28.710783 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.710770 2573 manager.go:319] Starting recovery of all containers Nov 03 15:07:28.716420 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.716381 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 15:07:28.722145 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.722117 2573 manager.go:324] Recovery completed Nov 03 15:07:28.726631 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.726618 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:07:28.730669 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.730640 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:07:28.730724 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.730679 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:07:28.730724 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.730689 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:07:28.731165 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.731146 2573 cpu_manager.go:226] "Starting" policy="none" Nov 03 15:07:28.731165 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.731157 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:07:28.731234 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.731157 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 15:07:28.731474 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.731460 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 15:07:28.732880 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.732862 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:07:28.732963 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.732915 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:07:28.732963 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.732936 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:07:28.733267 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.733255 2573 policy_none.go:50] "Start" Nov 03 15:07:28.733318 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.733273 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 15:07:28.733318 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.733286 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 15:07:28.734846 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.734833 2573 policy_none.go:44] "Start" Nov 03 15:07:28.760193 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.760157 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 15:07:28.763542 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.763531 2573 manager.go:341] "Starting Device Plugin manager" Nov 03 15:07:28.763609 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:28.763559 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 15:07:28.763609 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.763568 2573 server.go:87] "Starting device plugin registration server" Nov 03 15:07:28.763814 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.763803 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 15:07:28.763843 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.763817 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 15:07:28.763946 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.763923 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 15:07:28.764036 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.764009 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 15:07:28.764036 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.764020 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 15:07:28.764753 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:28.764730 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 15:07:28.764825 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:28.764777 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-207.us-west-1.compute.internal\" not found" Nov 03 15:07:28.843182 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.843057 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 15:07:28.845123 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.843761 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 15:07:28.845123 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.843797 2573 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 15:07:28.845123 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:28.843838 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 15:07:28.849081 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:28.849060 2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-207.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 15:07:28.849978 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:28.849060 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-207.us-west-1.compute.internal.18e4916cf1c343d4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-207.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-207.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 15:07:28.69890146 +0000 UTC m=+0.298370979,LastTimestamp:2026-11-03 15:07:28.69890146 +0000 UTC m=+0.298370979,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-207.us-west-1.compute.internal,}" Nov 03 15:07:28.864569 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.864551 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:07:28.865297 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.865279 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:07:28.865367 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.865314 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:07:28.865367 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.865330 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:07:28.865367 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.865357 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:28.898376 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.898308 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-207.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 15:07:28.923503 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.923483 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 15:07:28.923578 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.923555 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l46lz" Nov 03 15:07:28.945078 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.945057 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal"] Nov 03 15:07:28.945131 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.945110 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:07:28.945881 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.945868 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:07:28.945949 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.945896 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:07:28.945949 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.945912 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:07:28.947996 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.947983 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:07:28.948137 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.948123 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:28.948189 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.948153 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:07:28.948682 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.948649 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:07:28.948757 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.948691 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:07:28.948757 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.948705 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:07:28.948757 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.948727 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:07:28.948757 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.948744 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:07:28.948757 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.948757 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:07:28.950764 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.950749 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:28.950820 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.950779 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:07:28.951428 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.951411 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:07:28.951501 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.951438 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:07:28.951501 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:28.951448 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:07:28.975105 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:28.975087 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-207.us-west-1.compute.internal\" not found" node="ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:28.977584 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:28.977567 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-207.us-west-1.compute.internal\" not found" node="ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:29.011243 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.011220 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c495e56c3e03d09d80c3dcb0d6aed9eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal\" (UID: \"c495e56c3e03d09d80c3dcb0d6aed9eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:29.011318 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.011248 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c495e56c3e03d09d80c3dcb0d6aed9eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal\" (UID: \"c495e56c3e03d09d80c3dcb0d6aed9eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:29.011318 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.011271 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/abb209747bf3463921ac94593d5ab068-config\") pod \"kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal\" (UID: \"abb209747bf3463921ac94593d5ab068\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:29.011691 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.011678 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:07:29.060391 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.060372 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:07:29.110926 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.110909 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:07:29.111713 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.111695 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:07:29.111831 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.111714 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:29.111831 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.111720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c495e56c3e03d09d80c3dcb0d6aed9eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal\" (UID: \"c495e56c3e03d09d80c3dcb0d6aed9eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:29.111831 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:29.111736 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-207.us-west-1.compute.internal\": node \"ip-10-0-139-207.us-west-1.compute.internal\" not found" Nov 03 15:07:29.111831 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.111749 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/abb209747bf3463921ac94593d5ab068-config\") pod \"kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal\" (UID: \"abb209747bf3463921ac94593d5ab068\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:29.111831 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.111776 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c495e56c3e03d09d80c3dcb0d6aed9eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal\" (UID: \"c495e56c3e03d09d80c3dcb0d6aed9eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:29.111831 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.111811 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c495e56c3e03d09d80c3dcb0d6aed9eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal\" (UID: \"c495e56c3e03d09d80c3dcb0d6aed9eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:29.111831 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.111823 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c495e56c3e03d09d80c3dcb0d6aed9eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal\" (UID: \"c495e56c3e03d09d80c3dcb0d6aed9eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:29.111831 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.111828 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/abb209747bf3463921ac94593d5ab068-config\") pod \"kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal\" (UID: \"abb209747bf3463921ac94593d5ab068\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:29.210206 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.210148 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:29.273030 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.273007 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 15:07:29.274309 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.274297 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:29.279161 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.279148 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:29.281291 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.281277 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:07:29.334786 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.334766 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 15:07:29.642425 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.642348 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 15:07:29.642990 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.642507 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 15:07:29.642990 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.642521 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 15:07:29.642990 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.642521 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 15:07:29.642990 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:29.642522 2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a2ce9ff9192244e67a5502d625692119-459a7f00d10b34a9.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-207.us-west-1.compute.internal.18e4916cf3a85e26\": read tcp 10.0.139.207:36712->18.119.150.73:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-207.us-west-1.compute.internal.18e4916cf3a85e26 default 4932 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-207.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-207.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-207.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 15:07:28 +0000 UTC,LastTimestamp:2026-11-03 15:07:28.865337109 +0000 UTC m=+0.464806632,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-207.us-west-1.compute.internal,}" Nov 03 15:07:29.685296 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.685273 2573 apiserver.go:52] "Watching apiserver" Nov 03 15:07:29.709421 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.709402 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 15:07:29.928555 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.928473 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 15:02:30 +0000 UTC" deadline="2028-06-15 09:16:02.148617952 +0000 UTC" Nov 03 15:07:29.928555 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.928504 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14154h8m32.220117184s" Nov 03 15:07:29.952060 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.952036 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 15:07:29.952309 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.952294 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-lws9f","openshift-ovn-kubernetes/ovnkube-node-mmvs8","kube-system/konnectivity-agent-mh4np","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal","openshift-multus/multus-b5nrd","kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v","openshift-cluster-node-tuning-operator/tuned-6j7m4","openshift-multus/network-metrics-daemon-w4gnx","kube-system/global-pull-secret-syncer-7vw5n","openshift-image-registry/node-ca-kr9hf","openshift-network-operator/iptables-alerter-644l4","openshift-dns/node-resolver-rznqb","openshift-multus/multus-additional-cni-plugins-7k755"] Nov 03 15:07:29.954687 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.954664 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:29.954777 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:29.954730 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lws9f" podUID="50838860-ed04-41fa-b347-136f03437c18" Nov 03 15:07:29.956688 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.956677 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:29.958644 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.958628 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mh4np" Nov 03 15:07:29.960746 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.960734 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b5nrd" Nov 03 15:07:29.962953 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.962938 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:29.964973 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.964959 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:29.967018 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.967000 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:29.967083 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:29.967050 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4gnx" podUID="d2670401-d2a2-49fd-817a-18e04f38d7de" Nov 03 15:07:29.969123 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.969107 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:29.969188 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:29.969164 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vw5n" podUID="a1360ffe-4b52-4b77-8fbc-7a74c3b052a3" Nov 03 15:07:29.971191 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.971171 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kr9hf" Nov 03 15:07:29.973405 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.973389 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-644l4" Nov 03 15:07:29.975688 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.975674 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rznqb" Nov 03 15:07:29.977899 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:29.977885 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.436689 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.436673 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 15:07:31.437124 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.437105 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 15:07:31.437124 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.437108 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 15:07:31.437273 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.437152 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 15:07:31.437273 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.437251 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 15:07:31.437573 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.437555 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 15:07:31.437663 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.437644 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cjtnz\"" Nov 03 15:07:31.437856 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.437841 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 15:07:31.437912 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.437877 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 15:07:31.437953 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.437919 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 15:07:31.438103 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.438090 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4wckw\"" Nov 03 15:07:31.439774 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.439762 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dfz9p\"" Nov 03 15:07:31.440347 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.440327 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 15:07:31.440653 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.440512 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 15:07:31.440653 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.440592 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 15:07:31.440774 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.440716 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 15:07:31.442244 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.442224 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 15:07:31.442436 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.442419 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c432b1d0-d7d8-49c4-a26f-716b10424bc9-agent-certs\") pod \"konnectivity-agent-mh4np\" (UID: \"c432b1d0-d7d8-49c4-a26f-716b10424bc9\") " pod="kube-system/konnectivity-agent-mh4np" Nov 03 15:07:31.442517 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.442444 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 15:07:31.442557 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.442514 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-multus-socket-dir-parent\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.442606 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.442569 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z4c7\" (UniqueName: \"kubernetes.io/projected/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-kube-api-access-2z4c7\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.442698 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.442684 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fsh2r\"" Nov 03 15:07:31.442743 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.442716 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/444ebadb-41a3-4961-bc14-e2bc23ab2e93-tmp-dir\") pod \"node-resolver-rznqb\" (UID: \"444ebadb-41a3-4961-bc14-e2bc23ab2e93\") " pod="openshift-dns/node-resolver-rznqb" Nov 03 15:07:31.443089 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443053 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-log-socket\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.443150 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443104 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/731f41d1-b212-4b32-b0e1-b4d4188ed782-ovnkube-config\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.443150 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443128 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-device-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.443226 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443165 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:31.443226 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443200 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjgmv\" (UniqueName: \"kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv\") pod \"network-check-target-lws9f\" (UID: \"50838860-ed04-41fa-b347-136f03437c18\") " pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:31.443285 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443246 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-sysctl-conf\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.443285 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443277 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.443340 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443301 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-socket-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.443340 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443326 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7m7w\" (UniqueName: \"kubernetes.io/projected/d2670401-d2a2-49fd-817a-18e04f38d7de-kube-api-access-p7m7w\") pod \"network-metrics-daemon-w4gnx\" (UID: \"d2670401-d2a2-49fd-817a-18e04f38d7de\") " pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:31.443398 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443360 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-run\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.443398 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443385 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/444ebadb-41a3-4961-bc14-e2bc23ab2e93-hosts-file\") pod \"node-resolver-rznqb\" (UID: \"444ebadb-41a3-4961-bc14-e2bc23ab2e93\") " pod="openshift-dns/node-resolver-rznqb" Nov 03 15:07:31.443454 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443420 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-run-ovn\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.443454 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-cnibin\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.443546 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443464 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-run-multus-certs\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.443546 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443521 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-host\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.443661 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443549 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnrvl\" (UniqueName: \"kubernetes.io/projected/444ebadb-41a3-4961-bc14-e2bc23ab2e93-kube-api-access-wnrvl\") pod \"node-resolver-rznqb\" (UID: \"444ebadb-41a3-4961-bc14-e2bc23ab2e93\") " pod="openshift-dns/node-resolver-rznqb" Nov 03 15:07:31.443661 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443573 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-kubelet\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.443661 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443609 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-cni-netd\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.443661 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443647 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs\") pod \"network-metrics-daemon-w4gnx\" (UID: \"d2670401-d2a2-49fd-817a-18e04f38d7de\") " pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:31.443863 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d-host\") pod \"node-ca-kr9hf\" (UID: \"bda5e742-65f0-4b52-a3d9-b6e3a222ba4d\") " pod="openshift-image-registry/node-ca-kr9hf" Nov 03 15:07:31.443863 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443715 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/731f41d1-b212-4b32-b0e1-b4d4188ed782-ovnkube-script-lib\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.443863 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443742 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c432b1d0-d7d8-49c4-a26f-716b10424bc9-konnectivity-ca\") pod \"konnectivity-agent-mh4np\" (UID: \"c432b1d0-d7d8-49c4-a26f-716b10424bc9\") " pod="kube-system/konnectivity-agent-mh4np" Nov 03 15:07:31.443863 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443768 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-kubelet-config\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:31.443863 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443800 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d399188c-4d8b-4769-b212-bb2b2c485376-multus-daemon-config\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.443863 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443824 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-lib-modules\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.444029 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443875 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4b58f2a-a421-45cc-ba47-3b7519d9d074-cnibin\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.444029 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443911 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-registration-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.444029 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443933 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-system-cni-dir\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.444029 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443949 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-run-netns\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.444029 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443963 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-etc-kubernetes\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.444029 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.443977 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-tmp\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.444029 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444002 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-run-systemd\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.444029 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444023 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-var-lib-openvswitch\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.444375 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444037 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d399188c-4d8b-4769-b212-bb2b2c485376-cni-binary-copy\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.444375 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444068 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-modprobe-d\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.444375 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444095 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-kubernetes\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.444375 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444121 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-node-log\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.444375 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444166 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-hostroot\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.444375 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444191 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-var-lib-cni-multus\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.444375 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444212 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-run-k8s-cni-cncf-io\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.444375 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444231 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-var-lib-kubelet\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.444375 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444260 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4b58f2a-a421-45cc-ba47-3b7519d9d074-system-cni-dir\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.444375 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444279 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtd4c\" (UniqueName: \"kubernetes.io/projected/a4b58f2a-a421-45cc-ba47-3b7519d9d074-kube-api-access-rtd4c\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.444375 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444295 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1bcd546-0077-45d1-b7bd-bad96f267ad6-iptables-alerter-script\") pod \"iptables-alerter-644l4\" (UID: \"a1bcd546-0077-45d1-b7bd-bad96f267ad6\") " pod="openshift-network-operator/iptables-alerter-644l4" Nov 03 15:07:31.444375 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444310 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1bcd546-0077-45d1-b7bd-bad96f267ad6-host-slash\") pod \"iptables-alerter-644l4\" (UID: \"a1bcd546-0077-45d1-b7bd-bad96f267ad6\") " pod="openshift-network-operator/iptables-alerter-644l4" Nov 03 15:07:31.444375 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444326 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4b58f2a-a421-45cc-ba47-3b7519d9d074-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.444375 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444367 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-cni-bin\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.444791 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444391 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d-serviceca\") pod \"node-ca-kr9hf\" (UID: \"bda5e742-65f0-4b52-a3d9-b6e3a222ba4d\") " pod="openshift-image-registry/node-ca-kr9hf" Nov 03 15:07:31.444791 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444409 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfrqv\" (UniqueName: \"kubernetes.io/projected/a1bcd546-0077-45d1-b7bd-bad96f267ad6-kube-api-access-dfrqv\") pod \"iptables-alerter-644l4\" (UID: \"a1bcd546-0077-45d1-b7bd-bad96f267ad6\") " pod="openshift-network-operator/iptables-alerter-644l4" Nov 03 15:07:31.444791 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444441 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4b58f2a-a421-45cc-ba47-3b7519d9d074-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.444791 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444458 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-etc-openvswitch\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.444791 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444472 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-run-ovn-kubernetes\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.444791 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444519 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-run-netns\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.444791 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444555 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-run-openvswitch\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.444791 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444604 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.444791 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444630 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-etc-selinux\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.444791 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444666 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw4n9\" (UniqueName: \"kubernetes.io/projected/d399188c-4d8b-4769-b212-bb2b2c485376-kube-api-access-lw4n9\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.444791 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444699 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-sys\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.444791 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444724 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-var-lib-kubelet\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.444791 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444780 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-multus-cni-dir\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.445143 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444808 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-var-lib-cni-bin\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.445143 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444834 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4b58f2a-a421-45cc-ba47-3b7519d9d074-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.445143 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444862 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-sysctl-d\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.445143 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444883 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4b58f2a-a421-45cc-ba47-3b7519d9d074-cni-binary-copy\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.445143 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444917 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-sys-fs\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.445143 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444941 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-dbus\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:31.445143 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444961 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-sysconfig\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.445143 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.444983 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-tuned\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.445143 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.445012 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-systemd-units\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.445143 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.445049 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-slash\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.445143 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.445075 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/731f41d1-b212-4b32-b0e1-b4d4188ed782-env-overrides\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.445143 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.445096 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92vrw\" (UniqueName: \"kubernetes.io/projected/731f41d1-b212-4b32-b0e1-b4d4188ed782-kube-api-access-92vrw\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.445143 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.445138 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbmc7\" (UniqueName: \"kubernetes.io/projected/c5c66479-e6b2-40cb-acc4-5f61db6a778a-kube-api-access-vbmc7\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.445508 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.445169 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvvr8\" (UniqueName: \"kubernetes.io/projected/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d-kube-api-access-gvvr8\") pod \"node-ca-kr9hf\" (UID: \"bda5e742-65f0-4b52-a3d9-b6e3a222ba4d\") " pod="openshift-image-registry/node-ca-kr9hf" Nov 03 15:07:31.445508 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.445198 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-multus-conf-dir\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.445508 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.445221 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-systemd\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.445508 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.445238 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/731f41d1-b212-4b32-b0e1-b4d4188ed782-ovn-node-metrics-cert\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.481601 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.481566 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 15:07:31.481703 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.481603 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 15:07:31.481703 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.481697 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kdt56\"" Nov 03 15:07:31.535970 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.535954 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 15:07:31.536058 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.536004 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5blqx" Nov 03 15:07:31.536863 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.536848 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 15:07:31.536987 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.536965 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 15:07:31.537078 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.536994 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 15:07:31.537078 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.537036 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 15:07:31.537159 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.537137 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 15:07:31.537159 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.537155 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 15:07:31.537230 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.537178 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 15:07:31.537230 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.537157 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 15:07:31.537230 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.537197 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6cpt6\"" Nov 03 15:07:31.537230 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.537209 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 15:07:31.537338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.537256 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 15:07:31.537338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.537317 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jpxkd\"" Nov 03 15:07:31.537338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.537325 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qsch2\"" Nov 03 15:07:31.537414 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.537346 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cblvv\"" Nov 03 15:07:31.537414 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.537353 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 15:07:31.546363 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546346 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-kubelet\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.546443 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546367 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-cni-netd\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.546443 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546406 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs\") pod \"network-metrics-daemon-w4gnx\" (UID: \"d2670401-d2a2-49fd-817a-18e04f38d7de\") " pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:31.546509 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546431 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d-host\") pod \"node-ca-kr9hf\" (UID: \"bda5e742-65f0-4b52-a3d9-b6e3a222ba4d\") " pod="openshift-image-registry/node-ca-kr9hf" Nov 03 15:07:31.546509 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546458 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-cni-netd\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.546509 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546471 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/731f41d1-b212-4b32-b0e1-b4d4188ed782-ovnkube-script-lib\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.546509 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546458 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-kubelet\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.546509 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546486 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c432b1d0-d7d8-49c4-a26f-716b10424bc9-konnectivity-ca\") pod \"konnectivity-agent-mh4np\" (UID: \"c432b1d0-d7d8-49c4-a26f-716b10424bc9\") " pod="kube-system/konnectivity-agent-mh4np" Nov 03 15:07:31.546689 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546507 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d-host\") pod \"node-ca-kr9hf\" (UID: \"bda5e742-65f0-4b52-a3d9-b6e3a222ba4d\") " pod="openshift-image-registry/node-ca-kr9hf" Nov 03 15:07:31.546689 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:31.546511 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:31.546689 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546522 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-kubelet-config\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:31.546689 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546545 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d399188c-4d8b-4769-b212-bb2b2c485376-multus-daemon-config\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.546689 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:31.546589 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs podName:d2670401-d2a2-49fd-817a-18e04f38d7de nodeName:}" failed. No retries permitted until 2026-11-03 15:07:32.046558396 +0000 UTC m=+2.219100639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs") pod "network-metrics-daemon-w4gnx" (UID: "d2670401-d2a2-49fd-817a-18e04f38d7de") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:31.546689 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546609 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-lib-modules\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.546689 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546622 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-kubelet-config\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:31.546689 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546635 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4b58f2a-a421-45cc-ba47-3b7519d9d074-cnibin\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.546689 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546666 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-registration-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.547101 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546717 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4b58f2a-a421-45cc-ba47-3b7519d9d074-cnibin\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.547101 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546764 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-registration-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.547101 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546775 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-system-cni-dir\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.547101 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546803 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-run-netns\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.547101 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546815 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-system-cni-dir\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.547101 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546837 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-etc-kubernetes\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.547101 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546882 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-tmp\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.547101 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546911 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-run-systemd\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.547101 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546923 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-etc-kubernetes\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.547101 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546943 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-var-lib-openvswitch\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.547101 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546968 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d399188c-4d8b-4769-b212-bb2b2c485376-cni-binary-copy\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.547101 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.546991 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-modprobe-d\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.547101 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547013 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-kubernetes\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.547101 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547035 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-node-log\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.547101 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547059 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-hostroot\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.547101 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547082 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-var-lib-cni-multus\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.547101 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547106 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-run-k8s-cni-cncf-io\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.547812 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547119 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c432b1d0-d7d8-49c4-a26f-716b10424bc9-konnectivity-ca\") pod \"konnectivity-agent-mh4np\" (UID: \"c432b1d0-d7d8-49c4-a26f-716b10424bc9\") " pod="kube-system/konnectivity-agent-mh4np" Nov 03 15:07:31.547812 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547128 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-var-lib-kubelet\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.547812 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547123 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d399188c-4d8b-4769-b212-bb2b2c485376-multus-daemon-config\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.547812 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547152 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4b58f2a-a421-45cc-ba47-3b7519d9d074-system-cni-dir\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.547812 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547191 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 15:07:31.547812 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547220 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-run-netns\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.547812 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547227 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-modprobe-d\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.547812 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547235 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtd4c\" (UniqueName: \"kubernetes.io/projected/a4b58f2a-a421-45cc-ba47-3b7519d9d074-kube-api-access-rtd4c\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.547812 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547257 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-var-lib-cni-multus\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.547812 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547263 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1bcd546-0077-45d1-b7bd-bad96f267ad6-iptables-alerter-script\") pod \"iptables-alerter-644l4\" (UID: \"a1bcd546-0077-45d1-b7bd-bad96f267ad6\") " pod="openshift-network-operator/iptables-alerter-644l4" Nov 03 15:07:31.547812 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547285 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1bcd546-0077-45d1-b7bd-bad96f267ad6-host-slash\") pod \"iptables-alerter-644l4\" (UID: \"a1bcd546-0077-45d1-b7bd-bad96f267ad6\") " pod="openshift-network-operator/iptables-alerter-644l4" Nov 03 15:07:31.547812 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547288 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-hostroot\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.547812 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547287 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4b58f2a-a421-45cc-ba47-3b7519d9d074-system-cni-dir\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.547812 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547306 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4b58f2a-a421-45cc-ba47-3b7519d9d074-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.547812 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547320 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-kubernetes\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.547812 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547264 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-run-systemd\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.547812 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547336 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-var-lib-kubelet\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.547812 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547444 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-var-lib-openvswitch\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.548567 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547568 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-lib-modules\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.548567 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547576 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-node-log\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.548567 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547620 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-cni-bin\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.548567 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547628 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1bcd546-0077-45d1-b7bd-bad96f267ad6-host-slash\") pod \"iptables-alerter-644l4\" (UID: \"a1bcd546-0077-45d1-b7bd-bad96f267ad6\") " pod="openshift-network-operator/iptables-alerter-644l4" Nov 03 15:07:31.548567 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547646 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d-serviceca\") pod \"node-ca-kr9hf\" (UID: \"bda5e742-65f0-4b52-a3d9-b6e3a222ba4d\") " pod="openshift-image-registry/node-ca-kr9hf" Nov 03 15:07:31.548567 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547693 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-run-k8s-cni-cncf-io\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.548567 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547721 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfrqv\" (UniqueName: \"kubernetes.io/projected/a1bcd546-0077-45d1-b7bd-bad96f267ad6-kube-api-access-dfrqv\") pod \"iptables-alerter-644l4\" (UID: \"a1bcd546-0077-45d1-b7bd-bad96f267ad6\") " pod="openshift-network-operator/iptables-alerter-644l4" Nov 03 15:07:31.548567 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547747 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4b58f2a-a421-45cc-ba47-3b7519d9d074-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.548567 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547797 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-etc-openvswitch\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.548567 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547838 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-run-ovn-kubernetes\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.548567 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547861 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-run-netns\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.548567 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-run-openvswitch\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.548567 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547927 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.548567 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547949 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-etc-selinux\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.548567 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw4n9\" (UniqueName: \"kubernetes.io/projected/d399188c-4d8b-4769-b212-bb2b2c485376-kube-api-access-lw4n9\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.548567 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.547992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-sys\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.548567 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.548016 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-var-lib-kubelet\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.549231 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.548054 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-run-netns\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.549231 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.548270 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.549231 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.548305 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-cni-bin\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.549231 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.548521 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-etc-openvswitch\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.549231 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.548555 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-run-ovn-kubernetes\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.549231 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.548609 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-run-openvswitch\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.549231 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.548660 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-sys\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.549231 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.548723 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1bcd546-0077-45d1-b7bd-bad96f267ad6-iptables-alerter-script\") pod \"iptables-alerter-644l4\" (UID: \"a1bcd546-0077-45d1-b7bd-bad96f267ad6\") " pod="openshift-network-operator/iptables-alerter-644l4" Nov 03 15:07:31.549231 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.548744 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-etc-selinux\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.549231 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.548929 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-var-lib-kubelet\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.549231 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.548977 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-multus-cni-dir\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.549231 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549048 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/731f41d1-b212-4b32-b0e1-b4d4188ed782-ovnkube-script-lib\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.549231 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549061 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-var-lib-cni-bin\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.549231 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549107 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-var-lib-cni-bin\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.549231 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549144 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4b58f2a-a421-45cc-ba47-3b7519d9d074-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.549231 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549181 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-sysctl-d\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.549231 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549182 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-multus-cni-dir\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.549231 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549214 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d-serviceca\") pod \"node-ca-kr9hf\" (UID: \"bda5e742-65f0-4b52-a3d9-b6e3a222ba4d\") " pod="openshift-image-registry/node-ca-kr9hf" Nov 03 15:07:31.549961 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549231 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4b58f2a-a421-45cc-ba47-3b7519d9d074-cni-binary-copy\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.549961 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549261 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d399188c-4d8b-4769-b212-bb2b2c485376-cni-binary-copy\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.549961 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549273 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-sys-fs\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.549961 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549283 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4b58f2a-a421-45cc-ba47-3b7519d9d074-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.549961 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549323 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-sys-fs\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.549961 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-dbus\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:31.549961 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-sysconfig\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.549961 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549384 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-tuned\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.549961 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549406 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-sysconfig\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.549961 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549415 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-systemd-units\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.549961 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549434 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-sysctl-d\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.549961 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549440 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-slash\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.549961 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549450 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-dbus\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:31.549961 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549469 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-slash\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.549961 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549472 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/731f41d1-b212-4b32-b0e1-b4d4188ed782-env-overrides\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.549961 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549488 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-systemd-units\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.549961 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549510 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92vrw\" (UniqueName: \"kubernetes.io/projected/731f41d1-b212-4b32-b0e1-b4d4188ed782-kube-api-access-92vrw\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.550733 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549537 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbmc7\" (UniqueName: \"kubernetes.io/projected/c5c66479-e6b2-40cb-acc4-5f61db6a778a-kube-api-access-vbmc7\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.550733 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549559 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvvr8\" (UniqueName: \"kubernetes.io/projected/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d-kube-api-access-gvvr8\") pod \"node-ca-kr9hf\" (UID: \"bda5e742-65f0-4b52-a3d9-b6e3a222ba4d\") " pod="openshift-image-registry/node-ca-kr9hf" Nov 03 15:07:31.550733 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-multus-conf-dir\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.550733 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549635 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-systemd\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.550733 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549660 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/731f41d1-b212-4b32-b0e1-b4d4188ed782-ovn-node-metrics-cert\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.550733 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549765 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c432b1d0-d7d8-49c4-a26f-716b10424bc9-agent-certs\") pod \"konnectivity-agent-mh4np\" (UID: \"c432b1d0-d7d8-49c4-a26f-716b10424bc9\") " pod="kube-system/konnectivity-agent-mh4np" Nov 03 15:07:31.550733 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549791 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-multus-socket-dir-parent\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.550733 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z4c7\" (UniqueName: \"kubernetes.io/projected/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-kube-api-access-2z4c7\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.550733 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549831 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/444ebadb-41a3-4961-bc14-e2bc23ab2e93-tmp-dir\") pod \"node-resolver-rznqb\" (UID: \"444ebadb-41a3-4961-bc14-e2bc23ab2e93\") " pod="openshift-dns/node-resolver-rznqb" Nov 03 15:07:31.550733 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549851 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-log-socket\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.550733 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/731f41d1-b212-4b32-b0e1-b4d4188ed782-ovnkube-config\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.550733 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549901 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-device-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.550733 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549923 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:31.550733 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549946 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjgmv\" (UniqueName: \"kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv\") pod \"network-check-target-lws9f\" (UID: \"50838860-ed04-41fa-b347-136f03437c18\") " pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:31.550733 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549968 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-sysctl-conf\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.550733 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.549987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.550733 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550023 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-socket-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.551469 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550041 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7m7w\" (UniqueName: \"kubernetes.io/projected/d2670401-d2a2-49fd-817a-18e04f38d7de-kube-api-access-p7m7w\") pod \"network-metrics-daemon-w4gnx\" (UID: \"d2670401-d2a2-49fd-817a-18e04f38d7de\") " pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:31.551469 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550055 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-run\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.551469 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/444ebadb-41a3-4961-bc14-e2bc23ab2e93-hosts-file\") pod \"node-resolver-rznqb\" (UID: \"444ebadb-41a3-4961-bc14-e2bc23ab2e93\") " pod="openshift-dns/node-resolver-rznqb" Nov 03 15:07:31.551469 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550074 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-systemd\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.551469 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550082 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-run-ovn\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.551469 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550119 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-run-ovn\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.551469 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550120 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-cnibin\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.551469 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550157 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-cnibin\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.551469 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550161 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-run-multus-certs\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.551469 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550190 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-host\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.551469 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550194 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-run-multus-certs\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.551469 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550219 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnrvl\" (UniqueName: \"kubernetes.io/projected/444ebadb-41a3-4961-bc14-e2bc23ab2e93-kube-api-access-wnrvl\") pod \"node-resolver-rznqb\" (UID: \"444ebadb-41a3-4961-bc14-e2bc23ab2e93\") " pod="openshift-dns/node-resolver-rznqb" Nov 03 15:07:31.551469 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550409 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-multus-conf-dir\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.551469 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550451 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-host\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.551469 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550496 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-log-socket\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.551469 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:31.550647 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:31.551469 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:31.550748 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret podName:a1360ffe-4b52-4b77-8fbc-7a74c3b052a3 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:32.050729228 +0000 UTC m=+2.223271449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret") pod "global-pull-secret-syncer-7vw5n" (UID: "a1360ffe-4b52-4b77-8fbc-7a74c3b052a3") : object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:31.551469 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550743 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4b58f2a-a421-45cc-ba47-3b7519d9d074-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.552353 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550790 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/444ebadb-41a3-4961-bc14-e2bc23ab2e93-tmp-dir\") pod \"node-resolver-rznqb\" (UID: \"444ebadb-41a3-4961-bc14-e2bc23ab2e93\") " pod="openshift-dns/node-resolver-rznqb" Nov 03 15:07:31.552353 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550813 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-device-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.552353 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550892 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-multus-socket-dir-parent\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.552353 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550949 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-run\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.552353 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.550992 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.552353 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.551051 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/731f41d1-b212-4b32-b0e1-b4d4188ed782-ovnkube-config\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.552353 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.551085 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/444ebadb-41a3-4961-bc14-e2bc23ab2e93-hosts-file\") pod \"node-resolver-rznqb\" (UID: \"444ebadb-41a3-4961-bc14-e2bc23ab2e93\") " pod="openshift-dns/node-resolver-rznqb" Nov 03 15:07:31.552353 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.551089 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-socket-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.552353 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.551120 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-tmp\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.552353 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.551223 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-sysctl-conf\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.552353 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.551381 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/731f41d1-b212-4b32-b0e1-b4d4188ed782-env-overrides\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.552353 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.551630 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4b58f2a-a421-45cc-ba47-3b7519d9d074-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.552353 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.551729 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4b58f2a-a421-45cc-ba47-3b7519d9d074-cni-binary-copy\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.552353 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.551898 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-tuned\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.553037 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.552893 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c432b1d0-d7d8-49c4-a26f-716b10424bc9-agent-certs\") pod \"konnectivity-agent-mh4np\" (UID: \"c432b1d0-d7d8-49c4-a26f-716b10424bc9\") " pod="kube-system/konnectivity-agent-mh4np" Nov 03 15:07:31.553338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.553322 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/731f41d1-b212-4b32-b0e1-b4d4188ed782-ovn-node-metrics-cert\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.608309 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.608259 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtd4c\" (UniqueName: \"kubernetes.io/projected/a4b58f2a-a421-45cc-ba47-3b7519d9d074-kube-api-access-rtd4c\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.609615 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.609569 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfrqv\" (UniqueName: \"kubernetes.io/projected/a1bcd546-0077-45d1-b7bd-bad96f267ad6-kube-api-access-dfrqv\") pod \"iptables-alerter-644l4\" (UID: \"a1bcd546-0077-45d1-b7bd-bad96f267ad6\") " pod="openshift-network-operator/iptables-alerter-644l4" Nov 03 15:07:31.609717 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.609617 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw4n9\" (UniqueName: \"kubernetes.io/projected/d399188c-4d8b-4769-b212-bb2b2c485376-kube-api-access-lw4n9\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.610019 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.610003 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92vrw\" (UniqueName: \"kubernetes.io/projected/731f41d1-b212-4b32-b0e1-b4d4188ed782-kube-api-access-92vrw\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.613100 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:31.613080 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 15:07:31.613208 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:31.613105 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 15:07:31.613208 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:31.613117 2573 projected.go:196] Error preparing data for projected volume kube-api-access-mjgmv for pod openshift-network-diagnostics/network-check-target-lws9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:31.613208 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:31.613191 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv podName:50838860-ed04-41fa-b347-136f03437c18 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:32.11317328 +0000 UTC m=+2.285715514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mjgmv" (UniqueName: "kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv") pod "network-check-target-lws9f" (UID: "50838860-ed04-41fa-b347-136f03437c18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:31.613511 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.613490 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z4c7\" (UniqueName: \"kubernetes.io/projected/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-kube-api-access-2z4c7\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.613836 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.613818 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvvr8\" (UniqueName: \"kubernetes.io/projected/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d-kube-api-access-gvvr8\") pod \"node-ca-kr9hf\" (UID: \"bda5e742-65f0-4b52-a3d9-b6e3a222ba4d\") " pod="openshift-image-registry/node-ca-kr9hf" Nov 03 15:07:31.613970 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.613954 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnrvl\" (UniqueName: \"kubernetes.io/projected/444ebadb-41a3-4961-bc14-e2bc23ab2e93-kube-api-access-wnrvl\") pod \"node-resolver-rznqb\" (UID: \"444ebadb-41a3-4961-bc14-e2bc23ab2e93\") " pod="openshift-dns/node-resolver-rznqb" Nov 03 15:07:31.614164 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.614145 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbmc7\" (UniqueName: \"kubernetes.io/projected/c5c66479-e6b2-40cb-acc4-5f61db6a778a-kube-api-access-vbmc7\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.614966 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.614947 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7m7w\" (UniqueName: \"kubernetes.io/projected/d2670401-d2a2-49fd-817a-18e04f38d7de-kube-api-access-p7m7w\") pod \"network-metrics-daemon-w4gnx\" (UID: \"d2670401-d2a2-49fd-817a-18e04f38d7de\") " pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:31.729698 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.729673 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:31.733636 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.733620 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mh4np" Nov 03 15:07:31.739014 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.738998 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b5nrd" Nov 03 15:07:31.754837 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:31.754806 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc432b1d0_d7d8_49c4_a26f_716b10424bc9.slice/crio-3aab0474cba67a37664ddec19c1ca92e40132c9743c3e14156f96f4bc78cf1d8 WatchSource:0}: Error finding container 3aab0474cba67a37664ddec19c1ca92e40132c9743c3e14156f96f4bc78cf1d8: Status 404 returned error can't find the container with id 3aab0474cba67a37664ddec19c1ca92e40132c9743c3e14156f96f4bc78cf1d8 Nov 03 15:07:31.755887 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:31.755856 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb209747bf3463921ac94593d5ab068.slice/crio-1fb72603852cc9ae8393bf638086357ec418a7b05a3b7b55940e48d760043fda WatchSource:0}: Error finding container 1fb72603852cc9ae8393bf638086357ec418a7b05a3b7b55940e48d760043fda: Status 404 returned error can't find the container with id 1fb72603852cc9ae8393bf638086357ec418a7b05a3b7b55940e48d760043fda Nov 03 15:07:31.756559 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:31.756538 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd399188c_4d8b_4769_b212_bb2b2c485376.slice/crio-388e8b39d688a877ee41ede298c1f77649bf960d2e1553e68440c38e4c1e2f2b WatchSource:0}: Error finding container 388e8b39d688a877ee41ede298c1f77649bf960d2e1553e68440c38e4c1e2f2b: Status 404 returned error can't find the container with id 388e8b39d688a877ee41ede298c1f77649bf960d2e1553e68440c38e4c1e2f2b Nov 03 15:07:31.757188 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:31.757163 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc495e56c3e03d09d80c3dcb0d6aed9eb.slice/crio-97b5a2edac4885dfb5539d31ff3f862082f6ae68c9dd2287ed3355d21b1f9b34 WatchSource:0}: Error finding container 97b5a2edac4885dfb5539d31ff3f862082f6ae68c9dd2287ed3355d21b1f9b34: Status 404 returned error can't find the container with id 97b5a2edac4885dfb5539d31ff3f862082f6ae68c9dd2287ed3355d21b1f9b34 Nov 03 15:07:31.757916 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:31.757851 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731f41d1_b212_4b32_b0e1_b4d4188ed782.slice/crio-9dc44f194a865b4de6099b9870895af6f86c614fa62b055ea45cd05a1c7236ae WatchSource:0}: Error finding container 9dc44f194a865b4de6099b9870895af6f86c614fa62b055ea45cd05a1c7236ae: Status 404 returned error can't find the container with id 9dc44f194a865b4de6099b9870895af6f86c614fa62b055ea45cd05a1c7236ae Nov 03 15:07:31.758210 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.758190 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:07:31.760664 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.760645 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:07:31.767355 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.767334 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:07:31.769303 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:31.769285 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c66479_e6b2_40cb_acc4_5f61db6a778a.slice/crio-f066b34840337a5ad16cc37e002806d750f7edcee3d7d0c001e16a246256a4c7 WatchSource:0}: Error finding container f066b34840337a5ad16cc37e002806d750f7edcee3d7d0c001e16a246256a4c7: Status 404 returned error can't find the container with id f066b34840337a5ad16cc37e002806d750f7edcee3d7d0c001e16a246256a4c7 Nov 03 15:07:31.772303 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.772283 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kr9hf" Nov 03 15:07:31.775803 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:31.775778 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb8d4e4_6660_4a11_afe1_dc2aacfc49d3.slice/crio-8e0063df45d58d685352e4acf662c2691b102c9de9747a576bf2c2eb03cbc907 WatchSource:0}: Error finding container 8e0063df45d58d685352e4acf662c2691b102c9de9747a576bf2c2eb03cbc907: Status 404 returned error can't find the container with id 8e0063df45d58d685352e4acf662c2691b102c9de9747a576bf2c2eb03cbc907 Nov 03 15:07:31.780054 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:31.780035 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda5e742_65f0_4b52_a3d9_b6e3a222ba4d.slice/crio-fb2f439494b471824a4064a7a89eb9ae8db5a8f81c684f2fc458248850f12011 WatchSource:0}: Error finding container fb2f439494b471824a4064a7a89eb9ae8db5a8f81c684f2fc458248850f12011: Status 404 returned error can't find the container with id fb2f439494b471824a4064a7a89eb9ae8db5a8f81c684f2fc458248850f12011 Nov 03 15:07:31.802834 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.802816 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-644l4" Nov 03 15:07:31.807322 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.807310 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rznqb" Nov 03 15:07:31.811445 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:31.811424 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bcd546_0077_45d1_b7bd_bad96f267ad6.slice/crio-98280c819a3f1438de693c00755a4f84c73029ef6bbd4ee7120ab19ba1ac08f6 WatchSource:0}: Error finding container 98280c819a3f1438de693c00755a4f84c73029ef6bbd4ee7120ab19ba1ac08f6: Status 404 returned error can't find the container with id 98280c819a3f1438de693c00755a4f84c73029ef6bbd4ee7120ab19ba1ac08f6 Nov 03 15:07:31.816695 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:31.816678 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444ebadb_41a3_4961_bc14_e2bc23ab2e93.slice/crio-315b3138a39b352d269d58545edc80e5a9662128626dc82d89883f952928a116 WatchSource:0}: Error finding container 315b3138a39b352d269d58545edc80e5a9662128626dc82d89883f952928a116: Status 404 returned error can't find the container with id 315b3138a39b352d269d58545edc80e5a9662128626dc82d89883f952928a116 Nov 03 15:07:31.833717 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.833702 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:07:31.841702 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:07:31.841682 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b58f2a_a421_45cc_ba47_3b7519d9d074.slice/crio-d09f60e86e23711002de8ce6d7de447031c9dcc34c921f2fc9d76176b745a152 WatchSource:0}: Error finding container d09f60e86e23711002de8ce6d7de447031c9dcc34c921f2fc9d76176b745a152: Status 404 returned error can't find the container with id d09f60e86e23711002de8ce6d7de447031c9dcc34c921f2fc9d76176b745a152 Nov 03 15:07:31.843336 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:31.843317 2573 kuberuntime_manager.go:1707] "Unhandled Error" err="init container egress-router-binary-copy start failed in pod multus-additional-cni-plugins-7k755_openshift-multus(a4b58f2a-a421-45cc-ba47-3b7519d9d074): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Nov 03 15:07:31.843412 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:31.843346 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-7k755" podUID="a4b58f2a-a421-45cc-ba47-3b7519d9d074" Nov 03 15:07:31.982778 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:31.982717 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:07:32.053551 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:32.053524 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:32.053690 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:32.053565 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs\") pod \"network-metrics-daemon-w4gnx\" (UID: \"d2670401-d2a2-49fd-817a-18e04f38d7de\") " pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:32.053728 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:32.053698 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:32.053759 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:32.053739 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs podName:d2670401-d2a2-49fd-817a-18e04f38d7de nodeName:}" failed. No retries permitted until 2026-11-03 15:07:33.053727659 +0000 UTC m=+3.226269880 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs") pod "network-metrics-daemon-w4gnx" (UID: "d2670401-d2a2-49fd-817a-18e04f38d7de") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:32.053759 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:32.053698 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:32.053824 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:32.053766 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret podName:a1360ffe-4b52-4b77-8fbc-7a74c3b052a3 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:33.053760913 +0000 UTC m=+3.226303134 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret") pod "global-pull-secret-syncer-7vw5n" (UID: "a1360ffe-4b52-4b77-8fbc-7a74c3b052a3") : object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:32.154649 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:32.154623 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjgmv\" (UniqueName: \"kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv\") pod \"network-check-target-lws9f\" (UID: \"50838860-ed04-41fa-b347-136f03437c18\") " pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:32.155273 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:32.154729 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 15:07:32.155273 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:32.154744 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 15:07:32.155273 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:32.154752 2573 projected.go:196] Error preparing data for projected volume kube-api-access-mjgmv for pod openshift-network-diagnostics/network-check-target-lws9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:32.155273 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:32.154791 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv podName:50838860-ed04-41fa-b347-136f03437c18 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:33.15477969 +0000 UTC m=+3.327321925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjgmv" (UniqueName: "kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv") pod "network-check-target-lws9f" (UID: "50838860-ed04-41fa-b347-136f03437c18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:32.276520 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:32.276430 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal" event={"ID":"abb209747bf3463921ac94593d5ab068","Type":"ContainerStarted","Data":"1fb72603852cc9ae8393bf638086357ec418a7b05a3b7b55940e48d760043fda"} Nov 03 15:07:32.277501 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:32.277482 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7k755" event={"ID":"a4b58f2a-a421-45cc-ba47-3b7519d9d074","Type":"ContainerStarted","Data":"d09f60e86e23711002de8ce6d7de447031c9dcc34c921f2fc9d76176b745a152"} Nov 03 15:07:32.278713 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:32.278693 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rznqb" event={"ID":"444ebadb-41a3-4961-bc14-e2bc23ab2e93","Type":"ContainerStarted","Data":"315b3138a39b352d269d58545edc80e5a9662128626dc82d89883f952928a116"} Nov 03 15:07:32.279319 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:32.279293 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:65a324fc19ef3da53cecb9dc13e0c07696c595996d3c07d1a5dfd427f6a61e6e\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-7k755" podUID="a4b58f2a-a421-45cc-ba47-3b7519d9d074" Nov 03 15:07:32.279740 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:32.279719 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kr9hf" event={"ID":"bda5e742-65f0-4b52-a3d9-b6e3a222ba4d","Type":"ContainerStarted","Data":"fb2f439494b471824a4064a7a89eb9ae8db5a8f81c684f2fc458248850f12011"} Nov 03 15:07:32.280687 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:32.280670 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" event={"ID":"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3","Type":"ContainerStarted","Data":"8e0063df45d58d685352e4acf662c2691b102c9de9747a576bf2c2eb03cbc907"} Nov 03 15:07:32.281661 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:32.281644 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b5nrd" event={"ID":"d399188c-4d8b-4769-b212-bb2b2c485376","Type":"ContainerStarted","Data":"388e8b39d688a877ee41ede298c1f77649bf960d2e1553e68440c38e4c1e2f2b"} Nov 03 15:07:32.282636 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:32.282619 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" event={"ID":"731f41d1-b212-4b32-b0e1-b4d4188ed782","Type":"ContainerStarted","Data":"9dc44f194a865b4de6099b9870895af6f86c614fa62b055ea45cd05a1c7236ae"} Nov 03 15:07:32.283514 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:32.283496 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" event={"ID":"c495e56c3e03d09d80c3dcb0d6aed9eb","Type":"ContainerStarted","Data":"97b5a2edac4885dfb5539d31ff3f862082f6ae68c9dd2287ed3355d21b1f9b34"} Nov 03 15:07:32.284354 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:32.284336 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-644l4" event={"ID":"a1bcd546-0077-45d1-b7bd-bad96f267ad6","Type":"ContainerStarted","Data":"98280c819a3f1438de693c00755a4f84c73029ef6bbd4ee7120ab19ba1ac08f6"} Nov 03 15:07:32.285288 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:32.285247 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" event={"ID":"c5c66479-e6b2-40cb-acc4-5f61db6a778a","Type":"ContainerStarted","Data":"f066b34840337a5ad16cc37e002806d750f7edcee3d7d0c001e16a246256a4c7"} Nov 03 15:07:32.286286 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:32.286269 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mh4np" event={"ID":"c432b1d0-d7d8-49c4-a26f-716b10424bc9","Type":"ContainerStarted","Data":"3aab0474cba67a37664ddec19c1ca92e40132c9743c3e14156f96f4bc78cf1d8"} Nov 03 15:07:32.333988 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:32.333966 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:07:32.539020 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:32.538931 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 15:02:31 +0000 UTC" deadline="2028-06-08 12:26:04.298489175 +0000 UTC" Nov 03 15:07:32.539020 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:32.538969 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13989h18m31.759522515s" Nov 03 15:07:32.658708 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:32.658670 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:07:33.062642 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:33.062564 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:33.062830 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:33.062681 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs\") pod \"network-metrics-daemon-w4gnx\" (UID: \"d2670401-d2a2-49fd-817a-18e04f38d7de\") " pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:33.062875 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:33.062836 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:33.062910 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:33.062893 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs podName:d2670401-d2a2-49fd-817a-18e04f38d7de nodeName:}" failed. No retries permitted until 2026-11-03 15:07:35.062875378 +0000 UTC m=+5.235417603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs") pod "network-metrics-daemon-w4gnx" (UID: "d2670401-d2a2-49fd-817a-18e04f38d7de") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:33.062996 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:33.062957 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:33.062996 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:33.062992 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret podName:a1360ffe-4b52-4b77-8fbc-7a74c3b052a3 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:35.062982027 +0000 UTC m=+5.235524250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret") pod "global-pull-secret-syncer-7vw5n" (UID: "a1360ffe-4b52-4b77-8fbc-7a74c3b052a3") : object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:33.163293 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:33.163254 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjgmv\" (UniqueName: \"kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv\") pod \"network-check-target-lws9f\" (UID: \"50838860-ed04-41fa-b347-136f03437c18\") " pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:33.163779 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:33.163429 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 15:07:33.163779 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:33.163450 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 15:07:33.163779 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:33.163463 2573 projected.go:196] Error preparing data for projected volume kube-api-access-mjgmv for pod openshift-network-diagnostics/network-check-target-lws9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:33.163779 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:33.163517 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv podName:50838860-ed04-41fa-b347-136f03437c18 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:35.163500666 +0000 UTC m=+5.336042905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjgmv" (UniqueName: "kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv") pod "network-check-target-lws9f" (UID: "50838860-ed04-41fa-b347-136f03437c18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:33.271400 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:33.271369 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:33.271570 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:33.271497 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vw5n" podUID="a1360ffe-4b52-4b77-8fbc-7a74c3b052a3" Nov 03 15:07:33.271653 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:33.271570 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:33.271703 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:33.271655 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lws9f" podUID="50838860-ed04-41fa-b347-136f03437c18" Nov 03 15:07:33.271760 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:33.271715 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:33.271818 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:33.271798 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4gnx" podUID="d2670401-d2a2-49fd-817a-18e04f38d7de" Nov 03 15:07:33.539757 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:33.539611 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 15:02:31 +0000 UTC" deadline="2028-05-16 23:54:07.516212968 +0000 UTC" Nov 03 15:07:33.539757 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:33.539673 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13448h46m33.976544187s" Nov 03 15:07:35.079086 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:35.079049 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:35.079547 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:35.079123 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs\") pod \"network-metrics-daemon-w4gnx\" (UID: \"d2670401-d2a2-49fd-817a-18e04f38d7de\") " pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:35.079547 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:35.079265 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:35.079547 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:35.079327 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs podName:d2670401-d2a2-49fd-817a-18e04f38d7de nodeName:}" failed. No retries permitted until 2026-11-03 15:07:39.079310514 +0000 UTC m=+9.251852741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs") pod "network-metrics-daemon-w4gnx" (UID: "d2670401-d2a2-49fd-817a-18e04f38d7de") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:35.079773 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:35.079743 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:35.079841 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:35.079801 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret podName:a1360ffe-4b52-4b77-8fbc-7a74c3b052a3 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:39.079778657 +0000 UTC m=+9.252320880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret") pod "global-pull-secret-syncer-7vw5n" (UID: "a1360ffe-4b52-4b77-8fbc-7a74c3b052a3") : object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:35.180127 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:35.180029 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjgmv\" (UniqueName: \"kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv\") pod \"network-check-target-lws9f\" (UID: \"50838860-ed04-41fa-b347-136f03437c18\") " pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:35.180314 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:35.180178 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 15:07:35.180314 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:35.180195 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 15:07:35.180314 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:35.180208 2573 projected.go:196] Error preparing data for projected volume kube-api-access-mjgmv for pod openshift-network-diagnostics/network-check-target-lws9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:35.180314 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:35.180263 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv podName:50838860-ed04-41fa-b347-136f03437c18 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:39.180246433 +0000 UTC m=+9.352788670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjgmv" (UniqueName: "kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv") pod "network-check-target-lws9f" (UID: "50838860-ed04-41fa-b347-136f03437c18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:35.271512 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:35.271394 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:35.271718 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:35.271525 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vw5n" podUID="a1360ffe-4b52-4b77-8fbc-7a74c3b052a3" Nov 03 15:07:35.271718 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:35.271605 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:35.271718 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:35.271687 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4gnx" podUID="d2670401-d2a2-49fd-817a-18e04f38d7de" Nov 03 15:07:35.271881 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:35.271728 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:35.271881 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:35.271807 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lws9f" podUID="50838860-ed04-41fa-b347-136f03437c18" Nov 03 15:07:37.271449 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:37.271183 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:37.271449 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:37.271430 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:37.271998 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:37.271388 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:37.271998 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:37.271542 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lws9f" podUID="50838860-ed04-41fa-b347-136f03437c18" Nov 03 15:07:37.271998 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:37.271632 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4gnx" podUID="d2670401-d2a2-49fd-817a-18e04f38d7de" Nov 03 15:07:37.271998 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:37.271718 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vw5n" podUID="a1360ffe-4b52-4b77-8fbc-7a74c3b052a3" Nov 03 15:07:37.322728 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:37.322677 2573 generic.go:356] "Generic (PLEG): container finished" podID="c495e56c3e03d09d80c3dcb0d6aed9eb" containerID="2c000505ce1e75e2cd4de3ecc4abb306f77debe887bab0f5ee3de41c7a43b924" exitCode=0 Nov 03 15:07:37.322904 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:37.322762 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" event={"ID":"c495e56c3e03d09d80c3dcb0d6aed9eb","Type":"ContainerDied","Data":"2c000505ce1e75e2cd4de3ecc4abb306f77debe887bab0f5ee3de41c7a43b924"} Nov 03 15:07:37.327856 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:37.327831 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mh4np" event={"ID":"c432b1d0-d7d8-49c4-a26f-716b10424bc9","Type":"ContainerStarted","Data":"a85849ad258c3d08f35435a5f4ae3fb73fb007d168475773e39115dc87ebbf1c"} Nov 03 15:07:37.334978 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:37.334944 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal" event={"ID":"abb209747bf3463921ac94593d5ab068","Type":"ContainerStarted","Data":"9695eaeda0651f87261d76e1f35e9ebd3b48cbfc715f475cd0cbf1427e5c7396"} Nov 03 15:07:37.663606 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:37.663007 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal" podStartSLOduration=7.662989071 podStartE2EDuration="7.662989071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:37.550113386 +0000 UTC m=+7.722655629" watchObservedRunningTime="2026-11-03 15:07:37.662989071 +0000 UTC m=+7.835531314" Nov 03 15:07:38.274170 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:38.274111 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mh4np" Nov 03 15:07:38.274702 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:38.274681 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mh4np" Nov 03 15:07:38.340777 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:38.340676 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" event={"ID":"c5c66479-e6b2-40cb-acc4-5f61db6a778a","Type":"ContainerStarted","Data":"623e951b798ec4710ea61c33921c909e6982f9d8587429e1e8f40b9e4cca9611"} Nov 03 15:07:38.343049 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:38.343015 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kr9hf" event={"ID":"bda5e742-65f0-4b52-a3d9-b6e3a222ba4d","Type":"ContainerStarted","Data":"62c03baecbe6d96dd96ffa789d89987c6c7bea72c87b6256450970559eb6c7af"} Nov 03 15:07:38.347509 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:38.347484 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" event={"ID":"c495e56c3e03d09d80c3dcb0d6aed9eb","Type":"ContainerStarted","Data":"19c3f6381f95dcae4f0fe525b3811c22d67773beacfd8bde7e3850cb76f307d4"} Nov 03 15:07:38.389100 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:38.388033 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mh4np" podStartSLOduration=3.941384615 podStartE2EDuration="8.388014569s" totalImagesPullingTime="4.446629954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:37.663597581 +0000 UTC m=+7.836139825" watchObservedRunningTime="2026-11-03 15:07:38.388014569 +0000 UTC m=+8.560556812" Nov 03 15:07:38.630169 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:38.630077 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kr9hf" podStartSLOduration=2.797205491 podStartE2EDuration="8.630058123s" totalImagesPullingTime="5.832852632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:38.505205326 +0000 UTC m=+8.677747570" watchObservedRunningTime="2026-11-03 15:07:38.630058123 +0000 UTC m=+8.802600369" Nov 03 15:07:39.119030 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:39.118992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:39.119189 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:39.119063 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs\") pod \"network-metrics-daemon-w4gnx\" (UID: \"d2670401-d2a2-49fd-817a-18e04f38d7de\") " pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:39.119189 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:39.119106 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:39.119189 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:39.119154 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:39.119189 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:39.119178 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret podName:a1360ffe-4b52-4b77-8fbc-7a74c3b052a3 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:47.119160035 +0000 UTC m=+17.291702268 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret") pod "global-pull-secret-syncer-7vw5n" (UID: "a1360ffe-4b52-4b77-8fbc-7a74c3b052a3") : object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:39.119385 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:39.119210 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs podName:d2670401-d2a2-49fd-817a-18e04f38d7de nodeName:}" failed. No retries permitted until 2026-11-03 15:07:47.119192825 +0000 UTC m=+17.291735061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs") pod "network-metrics-daemon-w4gnx" (UID: "d2670401-d2a2-49fd-817a-18e04f38d7de") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:39.220408 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:39.220372 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjgmv\" (UniqueName: \"kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv\") pod \"network-check-target-lws9f\" (UID: \"50838860-ed04-41fa-b347-136f03437c18\") " pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:39.220604 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:39.220570 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 15:07:39.220682 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:39.220614 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 15:07:39.220682 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:39.220629 2573 projected.go:196] Error preparing data for projected volume kube-api-access-mjgmv for pod openshift-network-diagnostics/network-check-target-lws9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:39.220788 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:39.220691 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv podName:50838860-ed04-41fa-b347-136f03437c18 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:47.22067351 +0000 UTC m=+17.393215735 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjgmv" (UniqueName: "kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv") pod "network-check-target-lws9f" (UID: "50838860-ed04-41fa-b347-136f03437c18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:39.272145 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:39.272107 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:39.272308 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:39.272252 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lws9f" podUID="50838860-ed04-41fa-b347-136f03437c18" Nov 03 15:07:39.272364 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:39.272339 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:39.272462 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:39.272440 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4gnx" podUID="d2670401-d2a2-49fd-817a-18e04f38d7de" Nov 03 15:07:39.272534 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:39.272512 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:39.272626 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:39.272603 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vw5n" podUID="a1360ffe-4b52-4b77-8fbc-7a74c3b052a3" Nov 03 15:07:39.350279 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:39.350217 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 15:07:40.134868 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:40.134839 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 15:07:40.194029 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:40.193947 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T15:07:40.134862202Z","UUID":"223b614f-4192-419c-ac63-539bbec0d60f","Handler":null,"Name":"","Endpoint":""} Nov 03 15:07:40.195364 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:40.195341 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 15:07:40.195501 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:40.195373 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 15:07:40.353944 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:40.353909 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b5nrd" event={"ID":"d399188c-4d8b-4769-b212-bb2b2c485376","Type":"ContainerStarted","Data":"d9d21cf90bd8c7cf436eb1cef3deb040cd2dc7f857bccab7d3b73124ab03a25e"} Nov 03 15:07:40.356147 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:40.356121 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" event={"ID":"c5c66479-e6b2-40cb-acc4-5f61db6a778a","Type":"ContainerStarted","Data":"4f2d0729821b8668a6fedf334ecee5484c7c3d06df1175d38d150c2a5b665434"} Nov 03 15:07:40.357845 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:40.357821 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rznqb" event={"ID":"444ebadb-41a3-4961-bc14-e2bc23ab2e93","Type":"ContainerStarted","Data":"9d6d002ee221ed8cd91c0b32c8f6b09bf73f1a5a350dc63d462b90d060d2255c"} Nov 03 15:07:40.470801 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:40.470683 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" podStartSLOduration=10.470665372 podStartE2EDuration="10.470665372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:38.630971255 +0000 UTC m=+8.803513499" watchObservedRunningTime="2026-11-03 15:07:40.470665372 +0000 UTC m=+10.643207615" Nov 03 15:07:40.470961 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:40.470804 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-b5nrd" podStartSLOduration=2.674103949 podStartE2EDuration="10.470795559s" totalImagesPullingTime="7.79669161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:40.469615761 +0000 UTC m=+10.642158006" watchObservedRunningTime="2026-11-03 15:07:40.470795559 +0000 UTC m=+10.643337803" Nov 03 15:07:41.271441 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:41.270972 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:41.271441 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:41.270972 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:41.271441 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:41.271093 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lws9f" podUID="50838860-ed04-41fa-b347-136f03437c18" Nov 03 15:07:41.271441 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:41.271122 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:41.271441 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:41.271241 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vw5n" podUID="a1360ffe-4b52-4b77-8fbc-7a74c3b052a3" Nov 03 15:07:41.271441 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:41.271313 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4gnx" podUID="d2670401-d2a2-49fd-817a-18e04f38d7de" Nov 03 15:07:42.367376 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:42.367193 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" event={"ID":"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3","Type":"ContainerStarted","Data":"1e878886a78a5ea3d19f17bb17a773cc24be9aa28a4d24fd2e9cde3c09dbe5da"} Nov 03 15:07:42.368861 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:42.368832 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-644l4" event={"ID":"a1bcd546-0077-45d1-b7bd-bad96f267ad6","Type":"ContainerStarted","Data":"487741a91ac789096e47e4500d018cc204992b298166fb83b4d145a607244b32"} Nov 03 15:07:42.371253 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:42.371229 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" event={"ID":"c5c66479-e6b2-40cb-acc4-5f61db6a778a","Type":"ContainerStarted","Data":"42429069519c827da272cb9d7c53592fdb0df0f18b10c0ad3df5f6f47ed2350d"} Nov 03 15:07:42.482743 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:42.482694 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-rznqb" podStartSLOduration=4.450688021 podStartE2EDuration="12.48268197s" totalImagesPullingTime="8.031993949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:40.583313366 +0000 UTC m=+10.755855620" watchObservedRunningTime="2026-11-03 15:07:42.48268197 +0000 UTC m=+12.655224212" Nov 03 15:07:42.483188 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:42.483162 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" podStartSLOduration=2.665718478 podStartE2EDuration="12.483157802s" totalImagesPullingTime="9.817439324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:42.482206645 +0000 UTC m=+12.654748887" watchObservedRunningTime="2026-11-03 15:07:42.483157802 +0000 UTC m=+12.655700044" Nov 03 15:07:42.598081 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:42.598020 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" podStartSLOduration=3.117417314 podStartE2EDuration="12.598011963s" totalImagesPullingTime="9.480594649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:42.597686475 +0000 UTC m=+12.770228718" watchObservedRunningTime="2026-11-03 15:07:42.598011963 +0000 UTC m=+12.770554205" Nov 03 15:07:42.711542 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:42.711449 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-644l4" podStartSLOduration=4.483864348 podStartE2EDuration="12.711435558s" totalImagesPullingTime="8.22757121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:42.710943932 +0000 UTC m=+12.883486175" watchObservedRunningTime="2026-11-03 15:07:42.711435558 +0000 UTC m=+12.883977801" Nov 03 15:07:43.271625 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:43.271576 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:43.271801 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:43.271632 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:43.271801 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:43.271694 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:43.271801 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:43.271717 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vw5n" podUID="a1360ffe-4b52-4b77-8fbc-7a74c3b052a3" Nov 03 15:07:43.271801 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:43.271761 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lws9f" podUID="50838860-ed04-41fa-b347-136f03437c18" Nov 03 15:07:43.271965 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:43.271866 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4gnx" podUID="d2670401-d2a2-49fd-817a-18e04f38d7de" Nov 03 15:07:45.271907 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:45.271869 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:45.272766 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:45.271922 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:45.272766 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:45.272038 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:45.272766 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:45.272055 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vw5n" podUID="a1360ffe-4b52-4b77-8fbc-7a74c3b052a3" Nov 03 15:07:45.272766 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:45.272122 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lws9f" podUID="50838860-ed04-41fa-b347-136f03437c18" Nov 03 15:07:45.272766 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:45.272185 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4gnx" podUID="d2670401-d2a2-49fd-817a-18e04f38d7de" Nov 03 15:07:46.380953 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:46.380799 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:07:46.381243 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:46.381219 2573 generic.go:356] "Generic (PLEG): container finished" podID="731f41d1-b212-4b32-b0e1-b4d4188ed782" containerID="a165e8f37384f6f0efa4f635a055af6815fb66e2e39fd0ab2a45edab5ef7b245" exitCode=1 Nov 03 15:07:46.381285 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:46.381271 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" event={"ID":"731f41d1-b212-4b32-b0e1-b4d4188ed782","Type":"ContainerStarted","Data":"f1a6c8775801bdb99d67b189ca689658bed117f3101f1f46f217408876aa5ed3"} Nov 03 15:07:46.381342 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:46.381331 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" event={"ID":"731f41d1-b212-4b32-b0e1-b4d4188ed782","Type":"ContainerStarted","Data":"29a2db5734fc2019830c4357b3912dec76ba4dfd3903e05d1c2f5b4e20d739ff"} Nov 03 15:07:46.381388 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:46.381363 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" event={"ID":"731f41d1-b212-4b32-b0e1-b4d4188ed782","Type":"ContainerDied","Data":"a165e8f37384f6f0efa4f635a055af6815fb66e2e39fd0ab2a45edab5ef7b245"} Nov 03 15:07:46.381421 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:46.381391 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" event={"ID":"731f41d1-b212-4b32-b0e1-b4d4188ed782","Type":"ContainerStarted","Data":"ce6a52a9d9aa8b411643d2b50512d3362c1256cba65b80993a99d060e152f1c8"} Nov 03 15:07:47.179718 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:47.179676 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:47.179898 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:47.179736 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs\") pod \"network-metrics-daemon-w4gnx\" (UID: \"d2670401-d2a2-49fd-817a-18e04f38d7de\") " pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:47.179898 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:47.179853 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:47.179990 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:47.179924 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret podName:a1360ffe-4b52-4b77-8fbc-7a74c3b052a3 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:03.179902929 +0000 UTC m=+33.352445165 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret") pod "global-pull-secret-syncer-7vw5n" (UID: "a1360ffe-4b52-4b77-8fbc-7a74c3b052a3") : object "kube-system"/"original-pull-secret" not registered Nov 03 15:07:47.179990 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:47.179858 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:47.180070 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:47.179992 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs podName:d2670401-d2a2-49fd-817a-18e04f38d7de nodeName:}" failed. No retries permitted until 2026-11-03 15:08:03.17997665 +0000 UTC m=+33.352518885 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs") pod "network-metrics-daemon-w4gnx" (UID: "d2670401-d2a2-49fd-817a-18e04f38d7de") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 15:07:47.271313 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:47.271132 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:47.271539 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:47.271220 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:47.271539 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:47.271407 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lws9f" podUID="50838860-ed04-41fa-b347-136f03437c18" Nov 03 15:07:47.271539 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:47.271227 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:47.271539 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:47.271472 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vw5n" podUID="a1360ffe-4b52-4b77-8fbc-7a74c3b052a3" Nov 03 15:07:47.271539 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:47.271531 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4gnx" podUID="d2670401-d2a2-49fd-817a-18e04f38d7de" Nov 03 15:07:47.280297 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:47.280276 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjgmv\" (UniqueName: \"kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv\") pod \"network-check-target-lws9f\" (UID: \"50838860-ed04-41fa-b347-136f03437c18\") " pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:47.280433 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:47.280418 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 15:07:47.280479 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:47.280436 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 15:07:47.280479 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:47.280446 2573 projected.go:196] Error preparing data for projected volume kube-api-access-mjgmv for pod openshift-network-diagnostics/network-check-target-lws9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:47.280540 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:47.280487 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv podName:50838860-ed04-41fa-b347-136f03437c18 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:03.280475205 +0000 UTC m=+33.453017441 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjgmv" (UniqueName: "kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv") pod "network-check-target-lws9f" (UID: "50838860-ed04-41fa-b347-136f03437c18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 15:07:47.385070 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:47.385050 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:07:47.385491 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:47.385414 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" event={"ID":"731f41d1-b212-4b32-b0e1-b4d4188ed782","Type":"ContainerStarted","Data":"0814316ccc72d514e90a6c65752c5309cacc7776e4b313b9a1d09f6eaf83e7ce"} Nov 03 15:07:47.385491 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:47.385488 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" event={"ID":"731f41d1-b212-4b32-b0e1-b4d4188ed782","Type":"ContainerStarted","Data":"4e2bea7fe6a350dcade551bb90e7fc3dc32fef9288d6e0eebe7d66bb1ce6de70"} Nov 03 15:07:48.389117 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:48.389085 2573 generic.go:356] "Generic (PLEG): container finished" podID="a4b58f2a-a421-45cc-ba47-3b7519d9d074" containerID="813bf54ab27f38f3fd68385dab613ef0c0dfe02c01129ab75b59d1ab8150afb3" exitCode=0 Nov 03 15:07:48.389627 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:48.389142 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7k755" event={"ID":"a4b58f2a-a421-45cc-ba47-3b7519d9d074","Type":"ContainerDied","Data":"813bf54ab27f38f3fd68385dab613ef0c0dfe02c01129ab75b59d1ab8150afb3"} Nov 03 15:07:49.271176 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:49.271143 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:49.271368 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:49.271253 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lws9f" podUID="50838860-ed04-41fa-b347-136f03437c18" Nov 03 15:07:49.271368 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:49.271274 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:49.271490 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:49.271380 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vw5n" podUID="a1360ffe-4b52-4b77-8fbc-7a74c3b052a3" Nov 03 15:07:49.271597 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:49.271571 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:49.271721 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:49.271667 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4gnx" podUID="d2670401-d2a2-49fd-817a-18e04f38d7de" Nov 03 15:07:49.393858 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:49.393829 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:07:49.394402 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:49.394219 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" event={"ID":"731f41d1-b212-4b32-b0e1-b4d4188ed782","Type":"ContainerStarted","Data":"e8eaa19049f65dbeb443555b9046eb1d16e674bcaa12232136a5eb69d623c322"} Nov 03 15:07:51.271475 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:51.271237 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:51.272139 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:51.271291 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:51.272139 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:51.271510 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4gnx" podUID="d2670401-d2a2-49fd-817a-18e04f38d7de" Nov 03 15:07:51.272139 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:51.271299 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:51.272139 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:51.271575 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lws9f" podUID="50838860-ed04-41fa-b347-136f03437c18" Nov 03 15:07:51.272139 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:51.271672 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vw5n" podUID="a1360ffe-4b52-4b77-8fbc-7a74c3b052a3" Nov 03 15:07:51.400017 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:51.399981 2573 generic.go:356] "Generic (PLEG): container finished" podID="a4b58f2a-a421-45cc-ba47-3b7519d9d074" containerID="9e3328274ee8bbf49d26cfba5a2b904568d7486ebbaeb801ea70cfc3888cc02b" exitCode=0 Nov 03 15:07:51.400182 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:51.400052 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7k755" event={"ID":"a4b58f2a-a421-45cc-ba47-3b7519d9d074","Type":"ContainerDied","Data":"9e3328274ee8bbf49d26cfba5a2b904568d7486ebbaeb801ea70cfc3888cc02b"} Nov 03 15:07:51.403420 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:51.403402 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:07:51.403763 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:51.403747 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" event={"ID":"731f41d1-b212-4b32-b0e1-b4d4188ed782","Type":"ContainerStarted","Data":"f9bb73c69ad5688ee5ce7dbe1bf3a5db36f72c1e405e6084a143c8206d577d06"} Nov 03 15:07:51.404015 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:51.403998 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:51.404075 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:51.404063 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:51.404118 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:51.404098 2573 scope.go:122] "RemoveContainer" containerID="a165e8f37384f6f0efa4f635a055af6815fb66e2e39fd0ab2a45edab5ef7b245" Nov 03 15:07:51.419324 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:51.419304 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:52.409520 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:52.409494 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:07:52.409990 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:52.409881 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" event={"ID":"731f41d1-b212-4b32-b0e1-b4d4188ed782","Type":"ContainerStarted","Data":"09525ed16dd3ae4a6dba1582242c5947681cceb06ba8cae50e4e243560db0965"} Nov 03 15:07:52.410144 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:52.410120 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:52.424900 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:52.424874 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:07:52.615008 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:52.614960 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" podStartSLOduration=8.372056591 podStartE2EDuration="22.614945339s" totalImagesPullingTime="14.242888748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:52.61430015 +0000 UTC m=+22.786842394" watchObservedRunningTime="2026-11-03 15:07:52.614945339 +0000 UTC m=+22.787487582" Nov 03 15:07:53.271703 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:53.271619 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:53.271854 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:53.271637 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:53.271854 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:53.271800 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:53.271981 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:53.271922 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lws9f" podUID="50838860-ed04-41fa-b347-136f03437c18" Nov 03 15:07:53.272077 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:53.272054 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vw5n" podUID="a1360ffe-4b52-4b77-8fbc-7a74c3b052a3" Nov 03 15:07:53.272157 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:53.272142 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4gnx" podUID="d2670401-d2a2-49fd-817a-18e04f38d7de" Nov 03 15:07:53.342166 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:53.342132 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7vw5n"] Nov 03 15:07:53.345202 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:53.345176 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w4gnx"] Nov 03 15:07:53.348053 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:53.348029 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lws9f"] Nov 03 15:07:53.414698 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:53.414666 2573 generic.go:356] "Generic (PLEG): container finished" podID="a4b58f2a-a421-45cc-ba47-3b7519d9d074" containerID="1442b279dad68058dbe4dfb180606f435a67474bcf4c897f1c2522519efc02f8" exitCode=0 Nov 03 15:07:53.415174 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:53.414747 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:53.415174 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:53.414755 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7k755" event={"ID":"a4b58f2a-a421-45cc-ba47-3b7519d9d074","Type":"ContainerDied","Data":"1442b279dad68058dbe4dfb180606f435a67474bcf4c897f1c2522519efc02f8"} Nov 03 15:07:53.415174 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:53.414861 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lws9f" podUID="50838860-ed04-41fa-b347-136f03437c18" Nov 03 15:07:53.417948 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:53.417923 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:53.418085 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:53.418048 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vw5n" podUID="a1360ffe-4b52-4b77-8fbc-7a74c3b052a3" Nov 03 15:07:53.418142 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:53.418115 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:53.418267 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:53.418244 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4gnx" podUID="d2670401-d2a2-49fd-817a-18e04f38d7de" Nov 03 15:07:54.104172 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:54.104143 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:07:54.340538 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:54.340498 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:07:54.826930 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:54.826901 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:07:55.013379 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:55.013351 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:07:55.271876 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:55.271843 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:55.271876 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:55.271876 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:55.272120 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:55.271876 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:55.272120 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:55.271973 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4gnx" podUID="d2670401-d2a2-49fd-817a-18e04f38d7de" Nov 03 15:07:55.272120 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:55.272027 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lws9f" podUID="50838860-ed04-41fa-b347-136f03437c18" Nov 03 15:07:55.272120 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:55.272092 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vw5n" podUID="a1360ffe-4b52-4b77-8fbc-7a74c3b052a3" Nov 03 15:07:55.421906 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:55.421733 2573 generic.go:356] "Generic (PLEG): container finished" podID="a4b58f2a-a421-45cc-ba47-3b7519d9d074" containerID="6279b122fa39faee7432f4c03150a2fcc4a3710e6affedaed44ec19e9e31300e" exitCode=0 Nov 03 15:07:55.422060 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:55.421801 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7k755" event={"ID":"a4b58f2a-a421-45cc-ba47-3b7519d9d074","Type":"ContainerDied","Data":"6279b122fa39faee7432f4c03150a2fcc4a3710e6affedaed44ec19e9e31300e"} Nov 03 15:07:55.822528 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:55.822499 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:07:56.029901 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:56.029866 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:07:57.271210 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:57.271173 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:57.271830 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:57.271232 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:57.271830 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:57.271335 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lws9f" podUID="50838860-ed04-41fa-b347-136f03437c18" Nov 03 15:07:57.271830 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:57.271342 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:57.271830 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:57.271399 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vw5n" podUID="a1360ffe-4b52-4b77-8fbc-7a74c3b052a3" Nov 03 15:07:57.271830 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:57.271496 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4gnx" podUID="d2670401-d2a2-49fd-817a-18e04f38d7de" Nov 03 15:07:57.430392 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:57.430349 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7k755" event={"ID":"a4b58f2a-a421-45cc-ba47-3b7519d9d074","Type":"ContainerStarted","Data":"15bc0f58e067166b9fd2776a3ff9d5d34b9458a4d4235f3ebf641b4aa4e63b14"} Nov 03 15:07:57.720979 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:57.720949 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:07:57.914528 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:57.914498 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:07:58.434746 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:58.434717 2573 generic.go:356] "Generic (PLEG): container finished" podID="a4b58f2a-a421-45cc-ba47-3b7519d9d074" containerID="15bc0f58e067166b9fd2776a3ff9d5d34b9458a4d4235f3ebf641b4aa4e63b14" exitCode=0 Nov 03 15:07:58.435298 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:58.434774 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7k755" event={"ID":"a4b58f2a-a421-45cc-ba47-3b7519d9d074","Type":"ContainerDied","Data":"15bc0f58e067166b9fd2776a3ff9d5d34b9458a4d4235f3ebf641b4aa4e63b14"} Nov 03 15:07:59.037601 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.037556 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:07:59.219788 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.219751 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:07:59.271749 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.271691 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:07:59.271749 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.271728 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:07:59.271870 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.271708 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:07:59.271870 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:59.271806 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lws9f" podUID="50838860-ed04-41fa-b347-136f03437c18" Nov 03 15:07:59.271953 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:59.271878 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4gnx" podUID="d2670401-d2a2-49fd-817a-18e04f38d7de" Nov 03 15:07:59.271992 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:59.271961 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vw5n" podUID="a1360ffe-4b52-4b77-8fbc-7a74c3b052a3" Nov 03 15:07:59.439936 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.439899 2573 generic.go:356] "Generic (PLEG): container finished" podID="a4b58f2a-a421-45cc-ba47-3b7519d9d074" containerID="6c33ae8b5a1798de3e82de952e48eb477f844ca5f9f9f330f1ad6d829f495c33" exitCode=0 Nov 03 15:07:59.439936 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.439938 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7k755" event={"ID":"a4b58f2a-a421-45cc-ba47-3b7519d9d074","Type":"ContainerDied","Data":"6c33ae8b5a1798de3e82de952e48eb477f844ca5f9f9f330f1ad6d829f495c33"} Nov 03 15:07:59.628351 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.628326 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeReady" Nov 03 15:07:59.628502 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.628434 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 15:07:59.736055 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.736026 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hmch9"] Nov 03 15:07:59.738875 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.738860 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hmch9" Nov 03 15:07:59.743264 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.743206 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kpr4x"] Nov 03 15:07:59.746165 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.746146 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kpr4x" Nov 03 15:07:59.792247 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.792187 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 15:07:59.792247 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.792219 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ncdvq\"" Nov 03 15:07:59.792471 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.792457 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 15:07:59.792678 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.792664 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 15:07:59.798998 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.798974 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hmch9"] Nov 03 15:07:59.806820 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.806797 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kpr4x"] Nov 03 15:07:59.869975 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.869953 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-tmp-dir\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:07:59.870092 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.869996 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-config-volume\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:07:59.870092 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.870019 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:07:59.870092 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.870073 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c87d997-baa8-434f-9ccb-97ade4abe46d-cert\") pod \"ingress-canary-hmch9\" (UID: \"6c87d997-baa8-434f-9ccb-97ade4abe46d\") " pod="openshift-ingress-canary/ingress-canary-hmch9" Nov 03 15:07:59.870242 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.870148 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctthh\" (UniqueName: \"kubernetes.io/projected/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-kube-api-access-ctthh\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:07:59.870242 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.870189 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr9c8\" (UniqueName: \"kubernetes.io/projected/6c87d997-baa8-434f-9ccb-97ade4abe46d-kube-api-access-jr9c8\") pod \"ingress-canary-hmch9\" (UID: \"6c87d997-baa8-434f-9ccb-97ade4abe46d\") " pod="openshift-ingress-canary/ingress-canary-hmch9" Nov 03 15:07:59.892668 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.892647 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 15:07:59.892744 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.892702 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zncwb\"" Nov 03 15:07:59.892966 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.892951 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 15:07:59.970541 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.970512 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-config-volume\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:07:59.970691 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.970550 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:07:59.970691 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.970573 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c87d997-baa8-434f-9ccb-97ade4abe46d-cert\") pod \"ingress-canary-hmch9\" (UID: \"6c87d997-baa8-434f-9ccb-97ade4abe46d\") " pod="openshift-ingress-canary/ingress-canary-hmch9" Nov 03 15:07:59.970691 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.970612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctthh\" (UniqueName: \"kubernetes.io/projected/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-kube-api-access-ctthh\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:07:59.970691 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.970641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr9c8\" (UniqueName: \"kubernetes.io/projected/6c87d997-baa8-434f-9ccb-97ade4abe46d-kube-api-access-jr9c8\") pod \"ingress-canary-hmch9\" (UID: \"6c87d997-baa8-434f-9ccb-97ade4abe46d\") " pod="openshift-ingress-canary/ingress-canary-hmch9" Nov 03 15:07:59.970691 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.970686 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-tmp-dir\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:07:59.970963 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:59.970692 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 15:07:59.970963 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:07:59.970753 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls podName:ee7e4109-c1b4-4074-9c1e-cb40533b8df4 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:00.470733197 +0000 UTC m=+30.643275437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls") pod "dns-default-kpr4x" (UID: "ee7e4109-c1b4-4074-9c1e-cb40533b8df4") : secret "dns-default-metrics-tls" not found Nov 03 15:07:59.971076 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.970980 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-tmp-dir\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:07:59.971158 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.971141 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-config-volume\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:07:59.974623 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:07:59.974604 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c87d997-baa8-434f-9ccb-97ade4abe46d-cert\") pod \"ingress-canary-hmch9\" (UID: \"6c87d997-baa8-434f-9ccb-97ade4abe46d\") " pod="openshift-ingress-canary/ingress-canary-hmch9" Nov 03 15:08:00.032642 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:00.032613 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr9c8\" (UniqueName: \"kubernetes.io/projected/6c87d997-baa8-434f-9ccb-97ade4abe46d-kube-api-access-jr9c8\") pod \"ingress-canary-hmch9\" (UID: \"6c87d997-baa8-434f-9ccb-97ade4abe46d\") " pod="openshift-ingress-canary/ingress-canary-hmch9" Nov 03 15:08:00.032742 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:00.032659 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctthh\" (UniqueName: \"kubernetes.io/projected/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-kube-api-access-ctthh\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:08:00.049908 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:00.049888 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hmch9" Nov 03 15:08:00.431494 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:00.429933 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hmch9"] Nov 03 15:08:00.434856 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:08:00.434827 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c87d997_baa8_434f_9ccb_97ade4abe46d.slice/crio-5cb9557f926affbf41b0581e6e1dd7deccf1bcb69ff7576d5853b478152d6832 WatchSource:0}: Error finding container 5cb9557f926affbf41b0581e6e1dd7deccf1bcb69ff7576d5853b478152d6832: Status 404 returned error can't find the container with id 5cb9557f926affbf41b0581e6e1dd7deccf1bcb69ff7576d5853b478152d6832 Nov 03 15:08:00.443076 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:00.443054 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hmch9" event={"ID":"6c87d997-baa8-434f-9ccb-97ade4abe46d","Type":"ContainerStarted","Data":"5cb9557f926affbf41b0581e6e1dd7deccf1bcb69ff7576d5853b478152d6832"} Nov 03 15:08:00.446062 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:00.446042 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7k755" event={"ID":"a4b58f2a-a421-45cc-ba47-3b7519d9d074","Type":"ContainerStarted","Data":"c64b57fafe62093db9c8da00040709e6e9a9ef35711635bb57ef2eaac0865b7f"} Nov 03 15:08:00.474655 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:00.474631 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:08:00.474790 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:00.474773 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 15:08:00.474860 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:00.474839 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls podName:ee7e4109-c1b4-4074-9c1e-cb40533b8df4 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:01.47482111 +0000 UTC m=+31.647363351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls") pod "dns-default-kpr4x" (UID: "ee7e4109-c1b4-4074-9c1e-cb40533b8df4") : secret "dns-default-metrics-tls" not found Nov 03 15:08:00.569711 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:00.569662 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7k755" podStartSLOduration=3.975981128 podStartE2EDuration="30.569648906s" totalImagesPullingTime="26.593667778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:30 +0000 UTC" imagePullSessionsCount=6 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:00.568744836 +0000 UTC m=+30.741287079" watchObservedRunningTime="2026-11-03 15:08:00.569648906 +0000 UTC m=+30.742191150" Nov 03 15:08:00.951392 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:00.951353 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mh4np" Nov 03 15:08:00.951597 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:00.951483 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 15:08:00.951971 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:00.951955 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mh4np" Nov 03 15:08:01.184052 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.184018 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:01.271461 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.271368 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:08:01.271461 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.271450 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:08:01.271707 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.271474 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:08:01.324667 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.324555 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 15:08:01.324667 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.324639 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 15:08:01.324869 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.324817 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-975kv\"" Nov 03 15:08:01.324918 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.324871 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 15:08:01.324962 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.324917 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 15:08:01.325004 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.324968 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qbvnh\"" Nov 03 15:08:01.325052 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.325017 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z77jb\"" Nov 03 15:08:01.376909 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.376875 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:01.467856 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.467810 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x575x"] Nov 03 15:08:01.471091 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.471065 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:01.482553 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.482530 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:08:01.482670 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:01.482647 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 15:08:01.482738 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:01.482726 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls podName:ee7e4109-c1b4-4074-9c1e-cb40533b8df4 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:03.482710019 +0000 UTC m=+33.655252241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls") pod "dns-default-kpr4x" (UID: "ee7e4109-c1b4-4074-9c1e-cb40533b8df4") : secret "dns-default-metrics-tls" not found Nov 03 15:08:01.528639 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.528522 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 15:08:01.528639 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.528549 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 15:08:01.528639 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.528555 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 15:08:01.528639 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.528555 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 15:08:01.528639 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.528619 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g9729\"" Nov 03 15:08:01.533288 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.533265 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x575x"] Nov 03 15:08:01.573790 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.573744 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:01.583155 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.583109 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3723d5ac-70f9-42db-a027-7f84c5d622fa-crio-socket\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:01.583303 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.583179 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:01.583303 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.583220 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbxw5\" (UniqueName: \"kubernetes.io/projected/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-api-access-tbxw5\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:01.583303 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.583248 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3723d5ac-70f9-42db-a027-7f84c5d622fa-data-volume\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:01.583573 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.583552 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:01.684402 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.684373 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbxw5\" (UniqueName: \"kubernetes.io/projected/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-api-access-tbxw5\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:01.684553 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.684406 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3723d5ac-70f9-42db-a027-7f84c5d622fa-data-volume\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:01.684553 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.684457 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:01.684553 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.684500 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3723d5ac-70f9-42db-a027-7f84c5d622fa-crio-socket\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:01.684730 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:01.684562 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 15:08:01.684730 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.684674 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:01.684835 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:01.684723 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-rbac-proxy-cm podName:3723d5ac-70f9-42db-a027-7f84c5d622fa nodeName:}" failed. No retries permitted until 2026-11-03 15:08:02.18469733 +0000 UTC m=+32.357239559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x575x" (UID: "3723d5ac-70f9-42db-a027-7f84c5d622fa") : configmap "kube-rbac-proxy" not found Nov 03 15:08:01.684835 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:01.684776 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 15:08:01.684835 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:01.684827 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls podName:3723d5ac-70f9-42db-a027-7f84c5d622fa nodeName:}" failed. No retries permitted until 2026-11-03 15:08:02.184815383 +0000 UTC m=+32.357357605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x575x" (UID: "3723d5ac-70f9-42db-a027-7f84c5d622fa") : secret "insights-runtime-extractor-tls" not found Nov 03 15:08:01.684988 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.684834 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3723d5ac-70f9-42db-a027-7f84c5d622fa-crio-socket\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:01.684988 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.684866 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3723d5ac-70f9-42db-a027-7f84c5d622fa-data-volume\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:01.712576 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.712539 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-hwzmv-gkbt4"] Nov 03 15:08:01.715920 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.715887 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:01.749450 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.749417 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbxw5\" (UniqueName: \"kubernetes.io/projected/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-api-access-tbxw5\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:01.760181 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.760143 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:01.770166 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.770133 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ltlwc\"" Nov 03 15:08:01.770330 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.770287 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 15:08:01.786212 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.786134 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhhfg\" (UniqueName: \"kubernetes.io/projected/450408b0-4d59-4e6d-a643-a2d1e15e459e-kube-api-access-fhhfg\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:01.786212 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.786188 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-service-ca-bundle\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:01.786466 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.786259 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/450408b0-4d59-4e6d-a643-a2d1e15e459e-archives-path\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:01.786466 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.786367 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-trusted-ca-bundle\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:01.821729 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.821685 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 15:08:01.822424 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.822398 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hwzmv-gkbt4"] Nov 03 15:08:01.887420 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.887386 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-trusted-ca-bundle\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:01.887612 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.887466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhhfg\" (UniqueName: \"kubernetes.io/projected/450408b0-4d59-4e6d-a643-a2d1e15e459e-kube-api-access-fhhfg\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:01.887612 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.887504 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-service-ca-bundle\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:01.888632 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.887551 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/450408b0-4d59-4e6d-a643-a2d1e15e459e-archives-path\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:01.889064 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.889040 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/450408b0-4d59-4e6d-a643-a2d1e15e459e-archives-path\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:01.889221 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.889202 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-service-ca-bundle\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:01.889362 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.889341 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-trusted-ca-bundle\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:01.950690 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:01.950630 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhhfg\" (UniqueName: \"kubernetes.io/projected/450408b0-4d59-4e6d-a643-a2d1e15e459e-kube-api-access-fhhfg\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:02.029217 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.029179 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:02.102009 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.101973 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:02.192356 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.192316 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:02.192533 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.192395 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:02.192533 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:02.192432 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 15:08:02.192533 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:02.192505 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 15:08:02.192669 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:02.192509 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-rbac-proxy-cm podName:3723d5ac-70f9-42db-a027-7f84c5d622fa nodeName:}" failed. No retries permitted until 2026-11-03 15:08:03.192492685 +0000 UTC m=+33.365034931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x575x" (UID: "3723d5ac-70f9-42db-a027-7f84c5d622fa") : configmap "kube-rbac-proxy" not found Nov 03 15:08:02.192669 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:02.192596 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls podName:3723d5ac-70f9-42db-a027-7f84c5d622fa nodeName:}" failed. No retries permitted until 2026-11-03 15:08:03.192563271 +0000 UTC m=+33.365105499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x575x" (UID: "3723d5ac-70f9-42db-a027-7f84c5d622fa") : secret "insights-runtime-extractor-tls" not found Nov 03 15:08:02.243997 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.243963 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-x8qvd"] Nov 03 15:08:02.247741 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.247724 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x8qvd" Nov 03 15:08:02.290822 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.290743 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:02.304807 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.304736 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7mf6v\"" Nov 03 15:08:02.304977 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.304865 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 15:08:02.304977 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.304919 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 15:08:02.306360 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.306336 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-x8qvd"] Nov 03 15:08:02.394696 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.394648 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zt9k\" (UniqueName: \"kubernetes.io/projected/4a425029-f78c-4331-bb2a-00a94f6305d3-kube-api-access-5zt9k\") pod \"migrator-66df5d496b-x8qvd\" (UID: \"4a425029-f78c-4331-bb2a-00a94f6305d3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x8qvd" Nov 03 15:08:02.406654 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.406613 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hwzmv-gkbt4"] Nov 03 15:08:02.410237 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:08:02.410194 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450408b0_4d59_4e6d_a643_a2d1e15e459e.slice/crio-e26ac501b180bf89482d47c694bb618b1414817437a6b6bf005586eaaeb9f23a WatchSource:0}: Error finding container e26ac501b180bf89482d47c694bb618b1414817437a6b6bf005586eaaeb9f23a: Status 404 returned error can't find the container with id e26ac501b180bf89482d47c694bb618b1414817437a6b6bf005586eaaeb9f23a Nov 03 15:08:02.451965 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.451924 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" event={"ID":"450408b0-4d59-4e6d-a643-a2d1e15e459e","Type":"ContainerStarted","Data":"e26ac501b180bf89482d47c694bb618b1414817437a6b6bf005586eaaeb9f23a"} Nov 03 15:08:02.487111 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.487072 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:02.495443 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.495414 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zt9k\" (UniqueName: \"kubernetes.io/projected/4a425029-f78c-4331-bb2a-00a94f6305d3-kube-api-access-5zt9k\") pod \"migrator-66df5d496b-x8qvd\" (UID: \"4a425029-f78c-4331-bb2a-00a94f6305d3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x8qvd" Nov 03 15:08:02.538540 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.538506 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-x79q9"] Nov 03 15:08:02.542306 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.542236 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" Nov 03 15:08:02.558329 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.558288 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zt9k\" (UniqueName: \"kubernetes.io/projected/4a425029-f78c-4331-bb2a-00a94f6305d3-kube-api-access-5zt9k\") pod \"migrator-66df5d496b-x8qvd\" (UID: \"4a425029-f78c-4331-bb2a-00a94f6305d3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x8qvd" Nov 03 15:08:02.559012 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.558984 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x8qvd" Nov 03 15:08:02.599011 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.598985 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 15:08:02.599479 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.599464 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pzzbk\"" Nov 03 15:08:02.604799 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.604773 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-x79q9"] Nov 03 15:08:02.645715 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.645680 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 15:08:02.645878 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.645779 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 15:08:02.645878 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.645778 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 15:08:02.697444 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.697401 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d17094e-0f10-4e31-9afd-53daabac8d2d-signing-key\") pod \"service-ca-5fffc97bdf-x79q9\" (UID: \"1d17094e-0f10-4e31-9afd-53daabac8d2d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" Nov 03 15:08:02.697630 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.697462 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d17094e-0f10-4e31-9afd-53daabac8d2d-signing-cabundle\") pod \"service-ca-5fffc97bdf-x79q9\" (UID: \"1d17094e-0f10-4e31-9afd-53daabac8d2d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" Nov 03 15:08:02.697630 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.697486 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9kjl\" (UniqueName: \"kubernetes.io/projected/1d17094e-0f10-4e31-9afd-53daabac8d2d-kube-api-access-z9kjl\") pod \"service-ca-5fffc97bdf-x79q9\" (UID: \"1d17094e-0f10-4e31-9afd-53daabac8d2d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" Nov 03 15:08:02.742410 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.742369 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:02.798036 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.797994 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d17094e-0f10-4e31-9afd-53daabac8d2d-signing-key\") pod \"service-ca-5fffc97bdf-x79q9\" (UID: \"1d17094e-0f10-4e31-9afd-53daabac8d2d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" Nov 03 15:08:02.798197 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.798063 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d17094e-0f10-4e31-9afd-53daabac8d2d-signing-cabundle\") pod \"service-ca-5fffc97bdf-x79q9\" (UID: \"1d17094e-0f10-4e31-9afd-53daabac8d2d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" Nov 03 15:08:02.798197 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.798094 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9kjl\" (UniqueName: \"kubernetes.io/projected/1d17094e-0f10-4e31-9afd-53daabac8d2d-kube-api-access-z9kjl\") pod \"service-ca-5fffc97bdf-x79q9\" (UID: \"1d17094e-0f10-4e31-9afd-53daabac8d2d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" Nov 03 15:08:02.799056 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.799033 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d17094e-0f10-4e31-9afd-53daabac8d2d-signing-cabundle\") pod \"service-ca-5fffc97bdf-x79q9\" (UID: \"1d17094e-0f10-4e31-9afd-53daabac8d2d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" Nov 03 15:08:02.800768 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.800741 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d17094e-0f10-4e31-9afd-53daabac8d2d-signing-key\") pod \"service-ca-5fffc97bdf-x79q9\" (UID: \"1d17094e-0f10-4e31-9afd-53daabac8d2d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" Nov 03 15:08:02.860659 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.860624 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9kjl\" (UniqueName: \"kubernetes.io/projected/1d17094e-0f10-4e31-9afd-53daabac8d2d-kube-api-access-z9kjl\") pod \"service-ca-5fffc97bdf-x79q9\" (UID: \"1d17094e-0f10-4e31-9afd-53daabac8d2d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" Nov 03 15:08:02.935085 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.935044 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-x8qvd"] Nov 03 15:08:02.938339 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:08:02.938308 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a425029_f78c_4331_bb2a_00a94f6305d3.slice/crio-25fe9a1c1f525d3103ec79a7f0cd2d692f8cf40fd7d43d856b752afd51e613b0 WatchSource:0}: Error finding container 25fe9a1c1f525d3103ec79a7f0cd2d692f8cf40fd7d43d856b752afd51e613b0: Status 404 returned error can't find the container with id 25fe9a1c1f525d3103ec79a7f0cd2d692f8cf40fd7d43d856b752afd51e613b0 Nov 03 15:08:02.938959 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:02.938931 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:03.156244 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:03.156167 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" Nov 03 15:08:03.201533 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:03.201489 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:03.201730 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:03.201555 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:08:03.201730 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:03.201604 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:03.201730 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:03.201635 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs\") pod \"network-metrics-daemon-w4gnx\" (UID: \"d2670401-d2a2-49fd-817a-18e04f38d7de\") " pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:08:03.201730 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:03.201647 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 15:08:03.201730 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:03.201726 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-rbac-proxy-cm podName:3723d5ac-70f9-42db-a027-7f84c5d622fa nodeName:}" failed. No retries permitted until 2026-11-03 15:08:05.201703629 +0000 UTC m=+35.374245874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x575x" (UID: "3723d5ac-70f9-42db-a027-7f84c5d622fa") : configmap "kube-rbac-proxy" not found Nov 03 15:08:03.201971 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:03.201756 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 15:08:03.201971 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:03.201808 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs podName:d2670401-d2a2-49fd-817a-18e04f38d7de nodeName:}" failed. No retries permitted until 2026-11-03 15:08:35.201792777 +0000 UTC m=+65.374335007 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs") pod "network-metrics-daemon-w4gnx" (UID: "d2670401-d2a2-49fd-817a-18e04f38d7de") : secret "metrics-daemon-secret" not found Nov 03 15:08:03.201971 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:03.201871 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 15:08:03.201971 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:03.201907 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls podName:3723d5ac-70f9-42db-a027-7f84c5d622fa nodeName:}" failed. No retries permitted until 2026-11-03 15:08:05.201896509 +0000 UTC m=+35.374438746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x575x" (UID: "3723d5ac-70f9-42db-a027-7f84c5d622fa") : secret "insights-runtime-extractor-tls" not found Nov 03 15:08:03.204215 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:03.204180 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:08:03.302519 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:03.302484 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjgmv\" (UniqueName: \"kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv\") pod \"network-check-target-lws9f\" (UID: \"50838860-ed04-41fa-b347-136f03437c18\") " pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:08:03.304921 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:03.304895 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjgmv\" (UniqueName: \"kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv\") pod \"network-check-target-lws9f\" (UID: \"50838860-ed04-41fa-b347-136f03437c18\") " pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:08:03.391338 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:03.391299 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:08:03.394481 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:03.394447 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:08:03.457545 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:03.457511 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x8qvd" event={"ID":"4a425029-f78c-4331-bb2a-00a94f6305d3","Type":"ContainerStarted","Data":"25fe9a1c1f525d3103ec79a7f0cd2d692f8cf40fd7d43d856b752afd51e613b0"} Nov 03 15:08:03.504927 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:03.504891 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:08:03.505384 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:03.505050 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 15:08:03.505384 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:03.505120 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls podName:ee7e4109-c1b4-4074-9c1e-cb40533b8df4 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:07.50509797 +0000 UTC m=+37.677640210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls") pod "dns-default-kpr4x" (UID: "ee7e4109-c1b4-4074-9c1e-cb40533b8df4") : secret "dns-default-metrics-tls" not found Nov 03 15:08:03.609522 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:03.609495 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-x79q9"] Nov 03 15:08:03.612896 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:08:03.612868 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d17094e_0f10_4e31_9afd_53daabac8d2d.slice/crio-bb4d4bf6d49634c2e167f8da314a2d9bef2b9a7d7e88eee0819cf8c51e2eabb7 WatchSource:0}: Error finding container bb4d4bf6d49634c2e167f8da314a2d9bef2b9a7d7e88eee0819cf8c51e2eabb7: Status 404 returned error can't find the container with id bb4d4bf6d49634c2e167f8da314a2d9bef2b9a7d7e88eee0819cf8c51e2eabb7 Nov 03 15:08:03.615560 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:03.615537 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:03.784704 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:03.784088 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7vw5n"] Nov 03 15:08:03.790491 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:08:03.790460 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1360ffe_4b52_4b77_8fbc_7a74c3b052a3.slice/crio-2285b8f8b99aca9be9d08f7c63ae9436674374fa975cd1bbcdb307f73bd59d5e WatchSource:0}: Error finding container 2285b8f8b99aca9be9d08f7c63ae9436674374fa975cd1bbcdb307f73bd59d5e: Status 404 returned error can't find the container with id 2285b8f8b99aca9be9d08f7c63ae9436674374fa975cd1bbcdb307f73bd59d5e Nov 03 15:08:03.811888 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:03.811854 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lws9f"] Nov 03 15:08:03.814923 ip-10-0-139-207 kubenswrapper[2573]: W1103 15:08:03.814881 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50838860_ed04_41fa_b347_136f03437c18.slice/crio-c7e27844aa379b5cc35bb0844c794a6d285a27f8501dad460469ca22aa4a56d4 WatchSource:0}: Error finding container c7e27844aa379b5cc35bb0844c794a6d285a27f8501dad460469ca22aa4a56d4: Status 404 returned error can't find the container with id c7e27844aa379b5cc35bb0844c794a6d285a27f8501dad460469ca22aa4a56d4 Nov 03 15:08:04.461884 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:04.461812 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lws9f" event={"ID":"50838860-ed04-41fa-b347-136f03437c18","Type":"ContainerStarted","Data":"c7e27844aa379b5cc35bb0844c794a6d285a27f8501dad460469ca22aa4a56d4"} Nov 03 15:08:04.463265 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:04.463234 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7vw5n" event={"ID":"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3","Type":"ContainerStarted","Data":"2285b8f8b99aca9be9d08f7c63ae9436674374fa975cd1bbcdb307f73bd59d5e"} Nov 03 15:08:04.464716 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:04.464687 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" event={"ID":"1d17094e-0f10-4e31-9afd-53daabac8d2d","Type":"ContainerStarted","Data":"bb4d4bf6d49634c2e167f8da314a2d9bef2b9a7d7e88eee0819cf8c51e2eabb7"} Nov 03 15:08:04.466385 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:04.466350 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hmch9" event={"ID":"6c87d997-baa8-434f-9ccb-97ade4abe46d","Type":"ContainerStarted","Data":"d80f270061167774d779d9056e1ed3df23d8e933644989ac7e506c7668ccdc0c"} Nov 03 15:08:05.221953 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:05.221877 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:05.222379 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:05.222052 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 15:08:05.222379 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:05.222129 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls podName:3723d5ac-70f9-42db-a027-7f84c5d622fa nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.222106768 +0000 UTC m=+39.394648994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x575x" (UID: "3723d5ac-70f9-42db-a027-7f84c5d622fa") : secret "insights-runtime-extractor-tls" not found Nov 03 15:08:05.222500 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:05.222479 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:05.222634 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:05.222613 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 15:08:05.222714 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:05.222661 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-rbac-proxy-cm podName:3723d5ac-70f9-42db-a027-7f84c5d622fa nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.222647669 +0000 UTC m=+39.395189895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x575x" (UID: "3723d5ac-70f9-42db-a027-7f84c5d622fa") : configmap "kube-rbac-proxy" not found Nov 03 15:08:05.472245 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:05.471967 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x8qvd" event={"ID":"4a425029-f78c-4331-bb2a-00a94f6305d3","Type":"ContainerStarted","Data":"2383b5792626ced87efd291555539ea1862bf78a6678ce2ce4a5bef5c6b5a66e"} Nov 03 15:08:05.606054 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:05.605997 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x8qvd" podStartSLOduration=1.640917257 podStartE2EDuration="3.605984714s" totalImagesPullingTime="1.965067457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:05.605640295 +0000 UTC m=+35.778182538" watchObservedRunningTime="2026-11-03 15:08:05.605984714 +0000 UTC m=+35.778526958" Nov 03 15:08:05.606485 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:05.606453 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hmch9" podStartSLOduration=3.475156915 podStartE2EDuration="6.606445266s" totalImagesPullingTime="3.131288351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:07:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:04.585551812 +0000 UTC m=+34.758094057" watchObservedRunningTime="2026-11-03 15:08:05.606445266 +0000 UTC m=+35.778987509" Nov 03 15:08:05.705768 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:05.705724 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:05.901139 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:05.901104 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:06.476684 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:06.476631 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" event={"ID":"450408b0-4d59-4e6d-a643-a2d1e15e459e","Type":"ContainerStarted","Data":"22d8d72aa349c9a661dce61170ccedcf5bc970fbb285922774eb1ac43e8c4856"} Nov 03 15:08:06.534840 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:06.534805 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:06.593758 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:06.593645 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" podStartSLOduration=2.6067886639999998 podStartE2EDuration="5.593626663s" totalImagesPullingTime="2.986837999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:06.593206382 +0000 UTC m=+36.765748639" watchObservedRunningTime="2026-11-03 15:08:06.593626663 +0000 UTC m=+36.766168907" Nov 03 15:08:06.874411 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:06.874332 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:07.485435 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:07.485390 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" event={"ID":"1d17094e-0f10-4e31-9afd-53daabac8d2d","Type":"ContainerStarted","Data":"ad75bd2843095fc546e316a08cbac9b7d8d633eca2e5356f1812cf578cbca532"} Nov 03 15:08:07.547349 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:07.547304 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:08:07.547527 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:07.547481 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 15:08:07.547604 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:07.547552 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls podName:ee7e4109-c1b4-4074-9c1e-cb40533b8df4 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:15.547532992 +0000 UTC m=+45.720075236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls") pod "dns-default-kpr4x" (UID: "ee7e4109-c1b4-4074-9c1e-cb40533b8df4") : secret "dns-default-metrics-tls" not found Nov 03 15:08:07.603680 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:07.603559 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" podStartSLOduration=1.889571137 podStartE2EDuration="5.603537205s" totalImagesPullingTime="3.713966068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:07.60261385 +0000 UTC m=+37.775156093" watchObservedRunningTime="2026-11-03 15:08:07.603537205 +0000 UTC m=+37.776079453" Nov 03 15:08:08.489334 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:08.489295 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lws9f" event={"ID":"50838860-ed04-41fa-b347-136f03437c18","Type":"ContainerStarted","Data":"4ac006aab2aab1d869784d0c9470a835f6654e5100eeb182f6b167cfa9cbd2ad"} Nov 03 15:08:08.608872 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:08.608756 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lws9f" podStartSLOduration=34.121602686 podStartE2EDuration="38.608740406s" totalImagesPullingTime="4.48713772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:08.608174866 +0000 UTC m=+38.780717111" watchObservedRunningTime="2026-11-03 15:08:08.608740406 +0000 UTC m=+38.781282649" Nov 03 15:08:08.756057 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:08.755964 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:09.221781 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:09.221694 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rznqb_444ebadb-41a3-4961-bc14-e2bc23ab2e93/dns-node-resolver/0.log" Nov 03 15:08:09.264504 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:09.263825 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:09.264504 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:09.263915 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:09.264504 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:09.263945 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 15:08:09.264504 ip-10-0-139-207 kubenswrapper[2573]: E1103 15:08:09.264015 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls podName:3723d5ac-70f9-42db-a027-7f84c5d622fa nodeName:}" failed. No retries permitted until 2026-11-03 15:08:17.26399753 +0000 UTC m=+47.436539775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x575x" (UID: "3723d5ac-70f9-42db-a027-7f84c5d622fa") : secret "insights-runtime-extractor-tls" not found Nov 03 15:08:09.264841 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:09.264814 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:09.271889 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:09.271860 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:09.457775 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:09.457741 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:09.493659 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:09.492023 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:08:09.676779 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:09.676743 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:09.917003 ip-10-0-139-207 kubenswrapper[2573]: I1103 15:08:09.916972 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:10.071506 ip-10-0-139-207 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 15:08:10.079004 ip-10-0-139-207 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 15:08:10.079171 ip-10-0-139-207 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 15:08:10.079226 ip-10-0-139-207 systemd[1]: kubelet.service: Consumed 2.427s CPU time. Nov 03 15:08:10.090322 ip-10-0-139-207 systemd[1]: Starting Kubernetes Kubelet... Nov 03 15:08:10.207173 ip-10-0-139-207 kubenswrapper[4557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 15:08:10.207173 ip-10-0-139-207 kubenswrapper[4557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 15:08:10.207173 ip-10-0-139-207 kubenswrapper[4557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 15:08:10.207173 ip-10-0-139-207 kubenswrapper[4557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 15:08:10.209624 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209514 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:08:10.209624 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209621 4557 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:08:10.209624 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209626 4557 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209643 4557 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209648 4557 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209652 4557 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209656 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209660 4557 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209666 4557 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209670 4557 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209674 4557 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209678 4557 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209682 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209687 4557 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209691 4557 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209695 4557 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209700 4557 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209704 4557 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209707 4557 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209711 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209716 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:08:10.209844 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209720 4557 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209724 4557 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209728 4557 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209732 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209736 4557 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209740 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209744 4557 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209749 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209754 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209759 4557 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209763 4557 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209770 4557 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209774 4557 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209778 4557 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209783 4557 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209788 4557 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209795 4557 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209801 4557 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209807 4557 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209811 4557 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:08:10.210751 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209816 4557 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209821 4557 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209825 4557 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209829 4557 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209834 4557 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209838 4557 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209851 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209855 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209861 4557 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209865 4557 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209871 4557 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209876 4557 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209881 4557 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209886 4557 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209890 4557 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209895 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209899 4557 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209904 4557 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209908 4557 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:08:10.211651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209913 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209917 4557 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209922 4557 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209927 4557 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209931 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209936 4557 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209941 4557 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209945 4557 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209949 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209953 4557 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209957 4557 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209962 4557 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209966 4557 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209971 4557 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209975 4557 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209979 4557 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209983 4557 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209988 4557 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209992 4557 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.209996 4557 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:08:10.212507 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210003 4557 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210009 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210014 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210019 4557 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210024 4557 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210028 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210033 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210037 4557 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210041 4557 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210045 4557 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210049 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210054 4557 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210059 4557 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210064 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210068 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210073 4557 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210077 4557 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210081 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210087 4557 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210092 4557 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:08:10.213450 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210096 4557 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210101 4557 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210105 4557 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210110 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210114 4557 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.210120 4557 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211421 4557 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211435 4557 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211440 4557 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211444 4557 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211450 4557 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211454 4557 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211459 4557 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211464 4557 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211468 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211473 4557 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211477 4557 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211482 4557 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211486 4557 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211490 4557 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:08:10.214390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211494 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211499 4557 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211503 4557 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211508 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211513 4557 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211518 4557 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211525 4557 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211529 4557 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211534 4557 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211538 4557 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211542 4557 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211548 4557 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211552 4557 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211557 4557 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211561 4557 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211566 4557 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211572 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211593 4557 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211598 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:08:10.215302 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211604 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211609 4557 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211613 4557 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211617 4557 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211621 4557 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211626 4557 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211630 4557 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211637 4557 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211641 4557 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211645 4557 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211649 4557 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211653 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211658 4557 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211662 4557 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211667 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211671 4557 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211675 4557 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211680 4557 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211684 4557 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211689 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:08:10.216129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211693 4557 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211697 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211702 4557 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211706 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211710 4557 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211715 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211719 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211724 4557 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211729 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211733 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211737 4557 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211743 4557 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211751 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211756 4557 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211761 4557 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211767 4557 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211779 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211784 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211789 4557 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:08:10.216718 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211794 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211798 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211802 4557 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211807 4557 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211811 4557 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211816 4557 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211820 4557 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211824 4557 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211828 4557 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211832 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211839 4557 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211844 4557 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211850 4557 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211854 4557 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211858 4557 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211863 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211867 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211871 4557 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211875 4557 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211885 4557 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:08:10.217198 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211890 4557 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211894 4557 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211898 4557 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211903 4557 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211907 4557 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211913 4557 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211917 4557 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211921 4557 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211925 4557 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211930 4557 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211934 4557 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211938 4557 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211943 4557 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.211947 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212901 4557 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212912 4557 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212919 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212925 4557 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212930 4557 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212935 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:08:10.217713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212939 4557 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212943 4557 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212948 4557 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212952 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212957 4557 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212961 4557 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212965 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212969 4557 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212974 4557 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212979 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212983 4557 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212988 4557 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212993 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.212997 4557 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213001 4557 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213006 4557 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213010 4557 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213017 4557 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213021 4557 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213026 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:08:10.218264 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213030 4557 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213035 4557 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213039 4557 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213043 4557 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213048 4557 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213052 4557 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213056 4557 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213060 4557 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213064 4557 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213068 4557 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213073 4557 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213077 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213082 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213086 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213091 4557 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213095 4557 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213099 4557 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213104 4557 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213108 4557 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:08:10.219135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213112 4557 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213117 4557 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213121 4557 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213126 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213130 4557 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213135 4557 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213139 4557 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213143 4557 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213147 4557 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213151 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213158 4557 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213162 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213170 4557 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213176 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213181 4557 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213186 4557 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213190 4557 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213194 4557 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213199 4557 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213203 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:08:10.219745 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213207 4557 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213212 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213216 4557 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213220 4557 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213224 4557 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213229 4557 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213233 4557 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213237 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213241 4557 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213246 4557 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213250 4557 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213257 4557 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213262 4557 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213267 4557 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213271 4557 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213275 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213280 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213284 4557 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213289 4557 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:08:10.220253 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213294 4557 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213298 4557 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213302 4557 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213308 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213312 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213317 4557 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213323 4557 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213327 4557 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213332 4557 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213336 4557 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213341 4557 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213345 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213350 4557 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213356 4557 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213360 4557 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213364 4557 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213368 4557 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213373 4557 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213377 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213381 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:08:10.220808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213386 4557 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.213390 4557 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213530 4557 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213548 4557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213557 4557 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213564 4557 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213571 4557 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213593 4557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213601 4557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213608 4557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213613 4557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213618 4557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213624 4557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213630 4557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213635 4557 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213643 4557 flags.go:64] FLAG: --cgroup-root="" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213648 4557 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213652 4557 flags.go:64] FLAG: --client-ca-file="" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213657 4557 flags.go:64] FLAG: --cloud-provider="external" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213662 4557 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213670 4557 flags.go:64] FLAG: --cluster-domain="" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213674 4557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213679 4557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213685 4557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 15:08:10.221314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213691 4557 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213699 4557 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213704 4557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213709 4557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213715 4557 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213719 4557 flags.go:64] FLAG: --contention-profiling="false" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213724 4557 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213730 4557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213735 4557 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213740 4557 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213747 4557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213752 4557 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213757 4557 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213762 4557 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213767 4557 flags.go:64] FLAG: --enable-server="true" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213771 4557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213779 4557 flags.go:64] FLAG: --event-burst="100" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213784 4557 flags.go:64] FLAG: --event-qps="50" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213789 4557 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213794 4557 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213799 4557 flags.go:64] FLAG: --eviction-hard="" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213806 4557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213811 4557 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213818 4557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213824 4557 flags.go:64] FLAG: --eviction-soft="" Nov 03 15:08:10.221989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213829 4557 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213834 4557 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213839 4557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213844 4557 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213849 4557 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213853 4557 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213858 4557 flags.go:64] FLAG: --feature-gates="" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213865 4557 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213872 4557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213877 4557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213882 4557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213887 4557 flags.go:64] FLAG: --healthz-port="10248" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213892 4557 flags.go:64] FLAG: --help="false" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213897 4557 flags.go:64] FLAG: --hostname-override="ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213902 4557 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213907 4557 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213913 4557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213918 4557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213924 4557 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213929 4557 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213934 4557 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213938 4557 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213943 4557 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 15:08:10.222686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213948 4557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213953 4557 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213958 4557 flags.go:64] FLAG: --kube-reserved="" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213963 4557 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213968 4557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213973 4557 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213982 4557 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213988 4557 flags.go:64] FLAG: --lock-file="" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213993 4557 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.213998 4557 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214003 4557 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214012 4557 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214017 4557 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214022 4557 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214027 4557 flags.go:64] FLAG: --logging-format="text" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214032 4557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214037 4557 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214043 4557 flags.go:64] FLAG: --manifest-url="" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214049 4557 flags.go:64] FLAG: --manifest-url-header="" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214057 4557 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214062 4557 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214069 4557 flags.go:64] FLAG: --max-pods="110" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214074 4557 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214078 4557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214083 4557 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 15:08:10.223252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214088 4557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214093 4557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214098 4557 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214104 4557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214115 4557 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214120 4557 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214124 4557 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214129 4557 flags.go:64] FLAG: --pod-cidr="" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214134 4557 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214139 4557 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214144 4557 flags.go:64] FLAG: --pods-per-core="0" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214149 4557 flags.go:64] FLAG: --port="10250" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214154 4557 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214158 4557 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0af2ebdf5a7bca374" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214167 4557 flags.go:64] FLAG: --qos-reserved="" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214172 4557 flags.go:64] FLAG: --read-only-port="10255" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214177 4557 flags.go:64] FLAG: --register-node="true" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214182 4557 flags.go:64] FLAG: --register-with-taints="" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214189 4557 flags.go:64] FLAG: --registry-burst="10" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214193 4557 flags.go:64] FLAG: --registry-qps="5" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214198 4557 flags.go:64] FLAG: --reserved-cpus="" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214203 4557 flags.go:64] FLAG: --reserved-memory="" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214210 4557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214215 4557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214223 4557 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214228 4557 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 15:08:10.223881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214233 4557 flags.go:64] FLAG: --runonce="false" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214238 4557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214243 4557 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214248 4557 flags.go:64] FLAG: --seccomp-default="false" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214252 4557 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214257 4557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214262 4557 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214268 4557 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214273 4557 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214278 4557 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214282 4557 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214287 4557 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214291 4557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214296 4557 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214301 4557 flags.go:64] FLAG: --system-cgroups="" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214306 4557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214314 4557 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214319 4557 flags.go:64] FLAG: --tls-cert-file="" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214324 4557 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214331 4557 flags.go:64] FLAG: --tls-min-version="" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214337 4557 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214343 4557 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214348 4557 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214353 4557 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214358 4557 flags.go:64] FLAG: --v="2" Nov 03 15:08:10.224533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214371 4557 flags.go:64] FLAG: --version="false" Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214379 4557 flags.go:64] FLAG: --vmodule="" Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214385 4557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.214390 4557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214622 4557 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214632 4557 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214637 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214642 4557 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214647 4557 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214651 4557 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214656 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214660 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214664 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214669 4557 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214674 4557 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214678 4557 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214682 4557 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214687 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214691 4557 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214695 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:08:10.225241 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214699 4557 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214704 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214708 4557 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214713 4557 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214717 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214721 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214725 4557 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214730 4557 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214734 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214740 4557 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214744 4557 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214748 4557 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214753 4557 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214759 4557 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214765 4557 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214770 4557 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214776 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214783 4557 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214788 4557 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:08:10.226057 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214792 4557 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214796 4557 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214801 4557 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214805 4557 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214809 4557 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214814 4557 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214818 4557 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214822 4557 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214827 4557 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214831 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214836 4557 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214840 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214844 4557 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214849 4557 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214853 4557 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214858 4557 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214862 4557 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214866 4557 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214871 4557 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214875 4557 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:08:10.226650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214880 4557 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214884 4557 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214888 4557 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214893 4557 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214897 4557 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214902 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214906 4557 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214911 4557 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214915 4557 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214919 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214925 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214930 4557 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214934 4557 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214938 4557 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214942 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214947 4557 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214951 4557 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214955 4557 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214960 4557 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214964 4557 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:08:10.227201 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214970 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214974 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214979 4557 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214984 4557 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214989 4557 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214993 4557 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.214997 4557 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215001 4557 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215007 4557 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215011 4557 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215015 4557 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215019 4557 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215024 4557 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215028 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215032 4557 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215036 4557 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215041 4557 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215045 4557 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215050 4557 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215054 4557 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:08:10.227764 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215058 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:08:10.228412 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215063 4557 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:08:10.228412 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215071 4557 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:08:10.228412 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215077 4557 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:08:10.228412 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215084 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:08:10.228412 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215088 4557 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:08:10.228412 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215092 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:08:10.228412 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215097 4557 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:08:10.228412 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215101 4557 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:08:10.228412 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215106 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:08:10.228412 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215110 4557 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:08:10.228412 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.215152 4557 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 15:08:10.228412 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215210 4557 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:08:10.228412 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215218 4557 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:08:10.228412 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215223 4557 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:08:10.228412 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215227 4557 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215232 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215236 4557 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215240 4557 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215244 4557 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215249 4557 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215253 4557 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215258 4557 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215262 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215266 4557 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215272 4557 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215277 4557 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215282 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215286 4557 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215291 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215296 4557 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215300 4557 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215307 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215311 4557 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:08:10.228880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215317 4557 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215322 4557 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215326 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215331 4557 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215336 4557 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215341 4557 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215345 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215349 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215354 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215358 4557 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215363 4557 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215367 4557 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215371 4557 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215375 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215380 4557 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215385 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215389 4557 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215393 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215397 4557 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215402 4557 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:08:10.229390 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215406 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215410 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215414 4557 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215419 4557 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215423 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215427 4557 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215431 4557 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215436 4557 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215441 4557 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215448 4557 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215457 4557 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215462 4557 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215469 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215474 4557 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215478 4557 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215483 4557 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215487 4557 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215492 4557 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215496 4557 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:08:10.229919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215500 4557 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215505 4557 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215509 4557 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215513 4557 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215517 4557 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215522 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215526 4557 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215530 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215534 4557 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215539 4557 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215543 4557 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215549 4557 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215554 4557 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215558 4557 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215562 4557 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215566 4557 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215571 4557 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215575 4557 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215595 4557 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215600 4557 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:08:10.230397 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215604 4557 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215608 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215614 4557 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215621 4557 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215626 4557 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215632 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215637 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215641 4557 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215646 4557 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215650 4557 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215654 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215658 4557 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215663 4557 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215667 4557 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215671 4557 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215675 4557 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215680 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215684 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215689 4557 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215694 4557 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:08:10.230909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215698 4557 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215702 4557 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215707 4557 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215711 4557 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.215718 4557 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.220530 4557 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.220549 4557 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.220568 4557 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.220573 4557 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220655 4557 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220661 4557 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220666 4557 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220669 4557 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220672 4557 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220675 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220678 4557 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220681 4557 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220684 4557 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220688 4557 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:08:10.231410 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220691 4557 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220694 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220697 4557 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220700 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220703 4557 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220705 4557 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220713 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220716 4557 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220719 4557 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220722 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220724 4557 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220727 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220730 4557 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220732 4557 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220735 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220738 4557 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220741 4557 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220744 4557 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220746 4557 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220749 4557 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:08:10.231965 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220752 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220755 4557 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220758 4557 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220760 4557 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220763 4557 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220766 4557 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220768 4557 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220771 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220773 4557 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220776 4557 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220779 4557 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220782 4557 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220784 4557 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220787 4557 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220789 4557 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220792 4557 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220795 4557 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220798 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220801 4557 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220804 4557 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:08:10.232522 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220807 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220810 4557 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220812 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220815 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220818 4557 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220821 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220824 4557 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220827 4557 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220829 4557 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220832 4557 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220834 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220837 4557 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220840 4557 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220843 4557 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220845 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220848 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220851 4557 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220853 4557 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220856 4557 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:08:10.233225 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220859 4557 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220862 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220864 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220867 4557 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220870 4557 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220873 4557 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220876 4557 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220878 4557 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220881 4557 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220883 4557 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220887 4557 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220889 4557 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220892 4557 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220896 4557 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220900 4557 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220903 4557 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220907 4557 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220909 4557 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220912 4557 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:08:10.233984 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220915 4557 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:08:10.234786 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220917 4557 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:08:10.234786 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220920 4557 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:08:10.234786 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220922 4557 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:08:10.234786 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220925 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:08:10.234786 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220928 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:08:10.234786 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220931 4557 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:08:10.234786 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220934 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:08:10.234786 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220936 4557 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:08:10.234786 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220939 4557 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:08:10.234786 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220942 4557 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:08:10.234786 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220945 4557 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:08:10.234786 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220947 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:08:10.234786 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220950 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:08:10.234786 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220953 4557 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:08:10.234786 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220956 4557 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:08:10.234786 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220958 4557 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:08:10.234786 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.220961 4557 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:08:10.235326 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.220989 4557 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 15:08:10.235326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221137 4557 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:08:10.235326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221142 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:08:10.235326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221145 4557 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:08:10.235326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221147 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:08:10.235326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221150 4557 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:08:10.235326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221154 4557 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:08:10.235326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221157 4557 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:08:10.235326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221160 4557 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:08:10.235326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221163 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:08:10.235326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221165 4557 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:08:10.235326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221168 4557 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:08:10.235326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221171 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:08:10.235326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221173 4557 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:08:10.235326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221176 4557 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:08:10.235326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221179 4557 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221181 4557 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221184 4557 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221187 4557 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221189 4557 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221192 4557 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221194 4557 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221197 4557 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221200 4557 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221203 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221205 4557 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221208 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221211 4557 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221213 4557 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221216 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221218 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221221 4557 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221223 4557 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221227 4557 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:08:10.235819 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221231 4557 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221234 4557 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221238 4557 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221241 4557 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221244 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221247 4557 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221250 4557 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221253 4557 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221255 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221258 4557 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221260 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221263 4557 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221266 4557 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221268 4557 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221270 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221273 4557 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221275 4557 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221278 4557 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221281 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221283 4557 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:08:10.236284 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221286 4557 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221288 4557 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221291 4557 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221293 4557 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221296 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221298 4557 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221301 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221304 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221306 4557 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221308 4557 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221311 4557 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221314 4557 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221316 4557 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221319 4557 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221322 4557 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221325 4557 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221328 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221331 4557 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221335 4557 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221337 4557 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:08:10.236891 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221340 4557 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221344 4557 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221347 4557 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221350 4557 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221352 4557 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221355 4557 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221358 4557 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221361 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221363 4557 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221366 4557 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221368 4557 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221371 4557 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221373 4557 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221376 4557 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221378 4557 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221381 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221383 4557 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221386 4557 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221388 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:08:10.237385 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221391 4557 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:08:10.237862 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221393 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:08:10.237862 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221396 4557 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:08:10.237862 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221399 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:08:10.237862 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221401 4557 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:08:10.237862 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221404 4557 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:08:10.237862 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221407 4557 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:08:10.237862 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221409 4557 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:08:10.237862 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221412 4557 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:08:10.237862 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221415 4557 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:08:10.237862 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221417 4557 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:08:10.237862 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221420 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:08:10.237862 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221423 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:08:10.237862 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221426 4557 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:08:10.237862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.221449 4557 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 15:08:10.237862 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221480 4557 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 15:08:10.237862 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221484 4557 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221487 4557 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221490 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221493 4557 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221496 4557 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221498 4557 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221501 4557 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221503 4557 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221506 4557 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221508 4557 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221511 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221514 4557 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221516 4557 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221519 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221521 4557 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221524 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221527 4557 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221530 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221532 4557 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221534 4557 feature_gate.go:386] unrecognized feature gate: Example Nov 03 15:08:10.238523 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221537 4557 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221539 4557 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221542 4557 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221544 4557 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221547 4557 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221549 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221552 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221555 4557 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221557 4557 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221560 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221563 4557 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221565 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221568 4557 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221570 4557 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221573 4557 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221576 4557 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221592 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221596 4557 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221601 4557 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221605 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 15:08:10.239326 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221609 4557 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221613 4557 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221616 4557 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221619 4557 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221621 4557 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221624 4557 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221630 4557 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221633 4557 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221635 4557 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221638 4557 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221641 4557 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221644 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221647 4557 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221650 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221652 4557 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221655 4557 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221657 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221660 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221662 4557 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 15:08:10.240266 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221665 4557 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221670 4557 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221674 4557 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221678 4557 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221681 4557 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221685 4557 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221687 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221690 4557 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221693 4557 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221695 4557 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221698 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221700 4557 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221703 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221706 4557 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221708 4557 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221711 4557 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221714 4557 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221716 4557 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221719 4557 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 15:08:10.241031 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221722 4557 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221725 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221728 4557 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221731 4557 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221733 4557 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221736 4557 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221738 4557 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221741 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221744 4557 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221747 4557 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221749 4557 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221752 4557 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221755 4557 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221757 4557 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221762 4557 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221764 4557 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221767 4557 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221771 4557 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221773 4557 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221776 4557 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 15:08:10.241909 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221778 4557 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 15:08:10.242853 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221781 4557 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 15:08:10.242853 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221784 4557 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 15:08:10.242853 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221786 4557 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 15:08:10.242853 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221791 4557 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 15:08:10.242853 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221795 4557 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 15:08:10.242853 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:10.221798 4557 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 15:08:10.242853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.221875 4557 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 15:08:10.242853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.222870 4557 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 15:08:10.242853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.222929 4557 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 15:08:10.242853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.223140 4557 server.go:1014] "Starting client certificate rotation" Nov 03 15:08:10.242853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.223255 4557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 15:08:10.242853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.223294 4557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 15:02:30 +0000 UTC" deadline="2028-04-10 16:55:14.754833073 +0000 UTC" Nov 03 15:08:10.242853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.223329 4557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12577h47m4.531505835s" Nov 03 15:08:10.242853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.223630 4557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 15:08:10.242853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.224771 4557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 15:08:10.242853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.226286 4557 log.go:25] "Validated CRI v1 runtime API" Nov 03 15:08:10.243555 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.227888 4557 log.go:25] "Validated CRI v1 image API" Nov 03 15:08:10.243555 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.228650 4557 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 15:08:10.243555 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.230180 4557 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 82aa9dbe-8509-4886-b478-f6689de7328d:/dev/nvme0n1p4 ef47aad3-8d74-4d87-ae3e-79887c57d394:/dev/nvme0n1p3] Nov 03 15:08:10.243736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.230203 4557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1fb72603852cc9ae8393bf638086357ec418a7b05a3b7b55940e48d760043fda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1fb72603852cc9ae8393bf638086357ec418a7b05a3b7b55940e48d760043fda/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2285b8f8b99aca9be9d08f7c63ae9436674374fa975cd1bbcdb307f73bd59d5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2285b8f8b99aca9be9d08f7c63ae9436674374fa975cd1bbcdb307f73bd59d5e/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25fe9a1c1f525d3103ec79a7f0cd2d692f8cf40fd7d43d856b752afd51e613b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25fe9a1c1f525d3103ec79a7f0cd2d692f8cf40fd7d43d856b752afd51e613b0/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/315b3138a39b352d269d58545edc80e5a9662128626dc82d89883f952928a116/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/315b3138a39b352d269d58545edc80e5a9662128626dc82d89883f952928a116/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/388e8b39d688a877ee41ede298c1f77649bf960d2e1553e68440c38e4c1e2f2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/388e8b39d688a877ee41ede298c1f77649bf960d2e1553e68440c38e4c1e2f2b/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3aab0474cba67a37664ddec19c1ca92e40132c9743c3e14156f96f4bc78cf1d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3aab0474cba67a37664ddec19c1ca92e40132c9743c3e14156f96f4bc78cf1d8/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5cb9557f926affbf41b0581e6e1dd7deccf1bcb69ff7576d5853b478152d6832/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5cb9557f926affbf41b0581e6e1dd7deccf1bcb69ff7576d5853b478152d6832/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97b5a2edac4885dfb5539d31ff3f862082f6ae68c9dd2287ed3355d21b1f9b34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97b5a2edac4885dfb5539d31ff3f862082f6ae68c9dd2287ed3355d21b1f9b34/userdata/shm major:0 minor:52 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98280c819a3f1438de693c00755a4f84c73029ef6bbd4ee7120ab19ba1ac08f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98280c819a3f1438de693c00755a4f84c73029ef6bbd4ee7120ab19ba1ac08f6/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9dc44f194a865b4de6099b9870895af6f86c614fa62b055ea45cd05a1c7236ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9dc44f194a865b4de6099b9870895af6f86c614fa62b055ea45cd05a1c7236ae/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb4d4bf6d49634c2e167f8da314a2d9bef2b9a7d7e88eee0819cf8c51e2eabb7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb4d4bf6d49634c2e167f8da314a2d9bef2b9a7d7e88eee0819cf8c51e2eabb7/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7e27844aa379b5cc35bb0844c794a6d285a27f8501dad460469ca22aa4a56d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7e27844aa379b5cc35bb0844c794a6d285a27f8501dad460469ca22aa4a56d4/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d09f60e86e23711002de8ce6d7de447031c9dcc34c921f2fc9d76176b745a152/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d09f60e86e23711002de8ce6d7de447031c9dcc34c921f2fc9d76176b745a152/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e26ac501b180bf89482d47c694bb618b1414817437a6b6bf005586eaaeb9f23a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e26ac501b180bf89482d47c694bb618b1414817437a6b6bf005586eaaeb9f23a/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f066b34840337a5ad16cc37e002806d750f7edcee3d7d0c001e16a246256a4c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f066b34840337a5ad16cc37e002806d750f7edcee3d7d0c001e16a246256a4c7/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb2f439494b471824a4064a7a89eb9ae8db5a8f81c684f2fc458248850f12011/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb2f439494b471824a4064a7a89eb9ae8db5a8f81c684f2fc458248850f12011/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d17094e-0f10-4e31-9afd-53daabac8d2d/volumes/kubernetes.io~projected/kube-api-access-z9kjl:{mountpoint:/var/lib/kubelet/pods/1d17094e-0f10-4e31-9afd-53daabac8d2d/volumes/kubernetes.io~projected/kube-api-access-z9kjl major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d17094e-0f10-4e31-9afd-53daabac8d2d/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/1d17094e-0f10-4e31-9afd-53daabac8d2d/volumes/kubernetes.io~secret/signing-key major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3723d5ac-70f9-42db-a027-7f84c5d622fa/volumes/kubernetes.io~projected/kube-api-access-tbxw5:{mountpoint:/var/lib/kubelet/pods/3723d5ac-70f9-42db-a027-7f84c5d622fa/volumes/kubernetes.io~projected/kube-api-access-tbxw5 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/444ebadb-41a3-4961-bc14-e2bc23ab2e93/volumes/kubernetes.io~projected/kube-api-access-wnrvl:{mountpoint:/var/lib/kubelet/pods/444ebadb-41a3-4961-bc14-e2bc23ab2e93/volumes/kubernetes.io~projected/kube-api-access-wnrvl major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/450408b0-4d59-4e6d-a643-a2d1e15e459e/volumes/kubernetes.io~projected/kube-api-access-fhhfg:{mountpoint:/var/lib/kubelet/pods/450408b0-4d59-4e6d-a643-a2d1e15e459e/volumes/kubernetes.io~projected/kube-api-access-fhhfg major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a425029-f78c-4331-bb2a-00a94f6305d3/volumes/kubernetes.io~projected/kube-api-access-5zt9k:{mountpoint:/var/lib/kubelet/pods/4a425029-f78c-4331-bb2a-00a94f6305d3/volumes/kubernetes.io~projected/kube-api-access-5zt9k major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50838860-ed04-41fa-b347-136f03437c18/volumes/kubernetes.io~projected/kube-api-access-mjgmv:{mountpoint:/var/lib/kubelet/pods/50838860-ed04-41fa-b347-136f03437c18/volumes/kubernetes.io~projected/kube-api-access-mjgmv major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c87d997-baa8-434f-9ccb-97ade4abe46d/volumes/kubernetes.io~projected/kube-api-access-jr9c8:{mountpoint:/var/lib/kubelet/pods/6c87d997-baa8-434f-9ccb-97ade4abe46d/volumes/kubernetes.io~projected/kube-api-access-jr9c8 major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c87d997-baa8-434f-9ccb-97ade4abe46d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6c87d997-baa8-434f-9ccb-97ade4abe46d/volumes/kubernetes.io~secret/cert major:0 minor:196 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/731f41d1-b212-4b32-b0e1-b4d4188ed782/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/731f41d1-b212-4b32-b0e1-b4d4188ed782/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/731f41d1-b212-4b32-b0e1-b4d4188ed782/volumes/kubernetes.io~projected/kube-api-access-92vrw:{mountpoint:/var/lib/kubelet/pods/731f41d1-b212-4b32-b0e1-b4d4188ed782/volumes/kubernetes.io~projected/kube-api-access-92vrw major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/731f41d1-b212-4b32-b0e1-b4d4188ed782/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/731f41d1-b212-4b32-b0e1-b4d4188ed782/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1bcd546-0077-45d1-b7bd-bad96f267ad6/volumes/kubernetes.io~projected/kube-api-access-dfrqv:{mountpoint:/var/lib/kubelet/pods/a1bcd546-0077-45d1-b7bd-bad96f267ad6/volumes/kubernetes.io~projected/kube-api-access-dfrqv major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4b58f2a-a421-45cc-ba47-3b7519d9d074/volumes/kubernetes.io~projected/kube-api-access-rtd4c:{mountpoint:/var/lib/kubelet/pods/a4b58f2a-a421-45cc-ba47-3b7519d9d074/volumes/kubernetes.io~projected/kube-api-access-rtd4c major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d/volumes/kubernetes.io~projected/kube-api-access-gvvr8:{mountpoint:/var/lib/kubelet/pods/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d/volumes/kubernetes.io~projected/kube-api-access-gvvr8 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c432b1d0-d7d8-49c4-a26f-716b10424bc9/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c432b1d0-d7d8-49c4-a26f-716b10424bc9/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5c66479-e6b2-40cb-acc4-5f61db6a778a/volumes/kubernetes.io~projected/kube-api-access-vbmc7:{mountpoint:/var/lib/kubelet/pods/c5c66479-e6b2-40cb-acc4-5f61db6a778a/volumes/kubernetes.io~projected/kube-api-access-vbmc7 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2670401-d2a2-49fd-817a-18e04f38d7de/volumes/kubernetes.io~projected/kube-api-access-p7m7w:{mountpoint:/var/lib/kubelet/pods/d2670401-d2a2-49fd-817a-18e04f38d7de/volumes/kubernetes.io~projected/kube-api-access-p7m7w major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d399188c-4d8b-4769-b212-bb2b2c485376/volumes/kubernetes.io~projected/kube-api-access-lw4n9:{mountpoint:/var/lib/kubelet/pods/d399188c-4d8b-4769-b212-bb2b2c485376/volumes/kubernetes.io~projected/kube-api-access-lw4n9 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3/volumes/kubernetes.io~projected/kube-api-access-2z4c7:{mountpoint:/var/lib/kubelet/pods/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3/volumes/kubernetes.io~projected/kube-api-access-2z4c7 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee7e4109-c1b4-4074-9c1e-cb40533b8df4/volumes/kubernetes.io~projected/kube-api-access-ctthh:{mountpoint:/var/lib/kubelet/pods/ee7e4109-c1b4-4074-9c1e-cb40533b8df4/volumes/kubernetes.io~projected/kube-api-access-ctthh major:0 minor:201 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/10db3835246eee452b1167f45d061ae509d6c78d46bde48398235d583945417f/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/cd8e953e88db413f323be1e86c1eaf9dbf43b71c7c6573528f2757a62953f68e/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0b263a66508503bbbd01c0d9268babf3a41380edd8f2249bc79851f470aa05e8/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/cf2fc30e7cf4a114bf4c160d9502adbb4ff527fa43d8accdbd4b6ccc29336206/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/e43dc91164536f6075029c533ebd2cb15b977640c8fafb5829f558f2d68d47b9/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/342bb5dbb8aa4823e53dae4dbf614a48fd7a1802eba065ae353a243ecf54ce86/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/dd6c6788d158a662b41fa386e9c7a181b4fb5fb8c76f63aca3753b44bd5119b2/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/60ceeaa0bfbd1f6b957af1c6f2ff0ea6acf06131cf7f39ed4e3a08fc1c3a20a0/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b600d89aff2b5783f8932017458d7a7d3c4ea0eff670207aad72bebe95627e71/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/165d0b93f74f2d031d24684013fedb19870f53a345ac9529e55c8e6696d1ad64/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/33227f800ca9ccff835589aeab38cc6eb8df2bb740afa8abf1f559dbd797c67f/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/9bec01e95a053d77ce37e39ba160531a2b2773cabf582043e011712d04ba31c2/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/d41a7e9760995fd0735ccd027f7cf089f842365d1286bcb436297b1b160a7499/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/fe48216a1c3278765ac220779b620fe2405558793239088f6d1cd2e8e3f6cb53/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ba5d67e7721dd2693740bf26acb628dfc1676c04b9929bcf7a343f4b6304a3ed/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/941d854c7f4d4891d12eaa11a5dbf051f300886a7d5306e5e4ddf212d3ea9465/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2fe2108b7417eaf366e6acff33b552b962ee79f2ee09552b6e6211be462705d3/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/07ff3d4ddcfecbb66d355e6d3b5c74e6658f4e66fdb6b02638bfa80c6eb620ca/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/bb06fc67fd015e467f44be41a8c402bd0474d4736dd9b1b4525fcc125db26252/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/57353c25ec1d2632ec96d6cc08e3ae514bdd9acf63e64b2857e4628b2f248a0a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/522d3bc2d2c8286a904a0e1ed3f1595ad6c6d948d627c3c2d30e5723aa06456a/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/0004a7eb9d747fbe44d563de501500e6403362560417f8d20d2f427b83022e95/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/c92705ad9482793ca944030c7182899ce36a121519c1d5655b26c1b21512d51e/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/c49ab7d60381f18cb3a64e60f8ac94ed2f012443b3298578b7ae55fdb48f35e0/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/f83e462b89a134a25429cb01ab86fdb4510c261444238c74e0399bcb905a93a2/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/d74ca378df0901eb1ddd5b94bbe5b5efd215ba8dcb07d048e0ee0359f0cab27f/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/e48f549f4a0c726b214a1bb2aa4fc3c4fd951c23676871ee2bc5a399388f43e9/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/ee49e5c96875533e9e382c61c3b79fc581a69a1ecac8d206df12717f48641770/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/e8829b810d0a5ba9ca1224332aa55e247634982c9ea9f0195263c42be740d214/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/cb6f4dd8b161c5a9f088b7bb7e670696f4d0183807fdb752df259be836c9238b/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/7f5861d47d3ad63a5682775cb4e7bb9f8f71ef0146fdf3dd3717782a8784ea56/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/0ae95cfd9e39457b90f90349951fb580e33862436f53a8c597e784eff872d7ce/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/51641641f1f8820b39b8892d4f9ff07143bc44c26b6866c06786fc3320310b62/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/911afbe0056f2bd12e354e849558a6b5b73bc716dd32bf5b83a8f1b4dcb0bf91/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/7bbdb526bc3be0b7ec94e85fa23469229d21b1934c50f3bc1606ac5308b6026c/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/e6b17ffef86702e460f880d7be98dee9d17dfc0f2876b210b668ecf8ba1e1ba4/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/e4f26ec3be823f489a55ffb64af29f2a32f2bc6d0c6e42626116cef25b37bcec/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/2455fe62c777e63cba8f3aef2955698f8b525510e0ecfbfee61fdb2d54b3d5ce/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/6dcf13744ace437acb44edb3c031d7e0e590bf46008bab4a4d4f62489d717f7c/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/51c6d9f76fd3dc38ede9b9450e8cbfe85178daf712c3c1995030f28ec2cf4be5/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/b71e781de8f9959ecafc2a384b036c3fb942fe4242f06417e61d8a310a216459/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/0e4f93c14af4828bd0e0e1a0884217c1e980e1d933358b12fe9c438c3a5215ca/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5bca789cb49a8c8085ee4eaa842251265780a279a30724e5e927e51d0ba49410/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.237543 4557 manager.go:217] Machine: {Timestamp:2026-11-03 15:08:10.237083582 +0000 UTC m=+0.080205115 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101905 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22caec4fb4a97bde5d9d91f7504d99 SystemUUID:ec22caec-4fb4-a97b-de5d-9d91f7504d99 BootID:28580130-b2ac-485b-b89b-c53f0282f264 Filesystems:[{Device:/run/containers/storage/overlay-containers/f066b34840337a5ad16cc37e002806d750f7edcee3d7d0c001e16a246256a4c7/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c87d997-baa8-434f-9ccb-97ade4abe46d/volumes/kubernetes.io~projected/kube-api-access-jr9c8 DeviceMajor:0 DeviceMinor:200 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3723d5ac-70f9-42db-a027-7f84c5d622fa/volumes/kubernetes.io~projected/kube-api-access-tbxw5 DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/450408b0-4d59-4e6d-a643-a2d1e15e459e/volumes/kubernetes.io~projected/kube-api-access-fhhfg DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d09f60e86e23711002de8ce6d7de447031c9dcc34c921f2fc9d76176b745a152/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e26ac501b180bf89482d47c694bb618b1414817437a6b6bf005586eaaeb9f23a/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/731f41d1-b212-4b32-b0e1-b4d4188ed782/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c432b1d0-d7d8-49c4-a26f-716b10424bc9/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6c87d997-baa8-434f-9ccb-97ade4abe46d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:196 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d/volumes/kubernetes.io~projected/kube-api-access-gvvr8 DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7e27844aa379b5cc35bb0844c794a6d285a27f8501dad460469ca22aa4a56d4/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97b5a2edac4885dfb5539d31ff3f862082f6ae68c9dd2287ed3355d21b1f9b34/userdata/shm DeviceMajor:0 DeviceMinor:52 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee7e4109-c1b4-4074-9c1e-cb40533b8df4/volumes/kubernetes.io~projected/kube-api-access-ctthh DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25fe9a1c1f525d3103ec79a7f0cd2d692f8cf40fd7d43d856b752afd51e613b0/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d17094e-0f10-4e31-9afd-53daabac8d2d/volumes/kubernetes.io~projected/kube-api-access-z9kjl DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a4b58f2a-a421-45cc-ba47-3b7519d9d074/volumes/kubernetes.io~projected/kube-api-access-rtd4c DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a1bcd546-0077-45d1-b7bd-bad96f267ad6/volumes/kubernetes.io~projected/kube-api-access-dfrqv DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98280c819a3f1438de693c00755a4f84c73029ef6bbd4ee7120ab19ba1ac08f6/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9dc44f194a865b4de6099b9870895af6f86c614fa62b055ea45cd05a1c7236ae/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb2f439494b471824a4064a7a89eb9ae8db5a8f81c684f2fc458248850f12011/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4a425029-f78c-4331-bb2a-00a94f6305d3/volumes/kubernetes.io~projected/kube-api-access-5zt9k DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2285b8f8b99aca9be9d08f7c63ae9436674374fa975cd1bbcdb307f73bd59d5e/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/444ebadb-41a3-4961-bc14-e2bc23ab2e93/volumes/kubernetes.io~projected/kube-api-access-wnrvl DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1fb72603852cc9ae8393bf638086357ec418a7b05a3b7b55940e48d760043fda/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/315b3138a39b352d269d58545edc80e5a9662128626dc82d89883f952928a116/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/731f41d1-b212-4b32-b0e1-b4d4188ed782/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d2670401-d2a2-49fd-817a-18e04f38d7de/volumes/kubernetes.io~projected/kube-api-access-p7m7w DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/388e8b39d688a877ee41ede298c1f77649bf960d2e1553e68440c38e4c1e2f2b/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d17094e-0f10-4e31-9afd-53daabac8d2d/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d399188c-4d8b-4769-b212-bb2b2c485376/volumes/kubernetes.io~projected/kube-api-access-lw4n9 DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/731f41d1-b212-4b32-b0e1-b4d4188ed782/volumes/kubernetes.io~projected/kube-api-access-92vrw DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/c5c66479-e6b2-40cb-acc4-5f61db6a778a/volumes/kubernetes.io~projected/kube-api-access-vbmc7 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3/volumes/kubernetes.io~projected/kube-api-access-2z4c7 DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3aab0474cba67a37664ddec19c1ca92e40132c9743c3e14156f96f4bc78cf1d8/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5cb9557f926affbf41b0581e6e1dd7deccf1bcb69ff7576d5853b478152d6832/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb4d4bf6d49634c2e167f8da314a2d9bef2b9a7d7e88eee0819cf8c51e2eabb7/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/50838860-ed04-41fa-b347-136f03437c18/volumes/kubernetes.io~projected/kube-api-access-mjgmv DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2285b8f8b99aca9 MacAddress:86:d0:b3:60:e7:a4 Speed:10000 Mtu:8901} {Name:25fe9a1c1f525d3 MacAddress:22:b9:89:71:10:70 Speed:10000 Mtu:8901} {Name:5cb9557f926affb MacAddress:0e:d0:0a:38:9e:1f Speed:10000 Mtu:8901} {Name:bb4d4bf6d49634c MacAddress:de:8b:f5:51:af:79 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:6d:15:c3:75:35 Speed:0 Mtu:9001} {Name:br-int MacAddress:f6:6e:48:61:1d:8f Speed:0 Mtu:8901} {Name:c7e27844aa379b5 MacAddress:76:8d:83:db:42:45 Speed:10000 Mtu:8901} {Name:e26ac501b180bf8 MacAddress:be:5b:48:b0:19:bf Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:6d:15:c3:75:35 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:6a:76:77:22:1c:6b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:06:4e:06:49:65:54 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.237805 4557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.237853 4557 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.238216 4557 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.238247 4557 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-207.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.238454 4557 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.238467 4557 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.238481 4557 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.238505 4557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.238654 4557 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.238885 4557 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.239024 4557 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.239071 4557 kubelet.go:493] "Attempting to sync node with API server" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.239091 4557 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.239106 4557 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.239119 4557 kubelet.go:405] "Adding apiserver pod source" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.239131 4557 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.239974 4557 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.239987 4557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.240530 4557 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.243514 4557 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.243526 4557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.243664 4557 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.243945 4557 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 15:08:10.243984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.243987 4557 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 15:08:10.245673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.244141 4557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 15:08:10.245673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.244158 4557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 15:08:10.245673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.244178 4557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 15:08:10.245673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.244188 4557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 15:08:10.245673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.244198 4557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 15:08:10.245673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.244208 4557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 15:08:10.245673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.244216 4557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 15:08:10.245673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.244226 4557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 15:08:10.245673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.244237 4557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 15:08:10.245673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.244248 4557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 15:08:10.245673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.244261 4557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 15:08:10.245673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.244275 4557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 15:08:10.245673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.244310 4557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 15:08:10.245673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.244322 4557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 15:08:10.248142 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.248125 4557 server.go:1263] "Started kubelet" Nov 03 15:08:10.248567 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.248478 4557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 15:08:10.248673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.248608 4557 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 15:08:10.248626 ip-10-0-139-207 systemd[1]: Started Kubernetes Kubelet. Nov 03 15:08:10.248868 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.248721 4557 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 15:08:10.249092 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.249071 4557 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 15:08:10.250083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.250066 4557 server.go:317] "Adding debug handlers to kubelet server" Nov 03 15:08:10.251847 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.251787 4557 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 15:08:10.252696 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.252485 4557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 15:08:10.252696 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.252550 4557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 15:02:31 +0000 UTC" deadline="2028-05-22 00:00:52.734091282 +0000 UTC" Nov 03 15:08:10.252696 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.252611 4557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13568h52m42.481484291s" Nov 03 15:08:10.252880 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.252805 4557 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 15:08:10.252880 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.252812 4557 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 15:08:10.252974 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.252932 4557 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.254029 4557 factory.go:55] Registering systemd factory Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.254051 4557 factory.go:223] Registration of the systemd container factory successfully Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:10.254218 4557 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-207.us-west-1.compute.internal\" not found" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.254328 4557 factory.go:153] Registering CRI-O factory Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.254342 4557 factory.go:223] Registration of the crio container factory successfully Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.254410 4557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.254432 4557 factory.go:103] Registering Raw factory Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.254449 4557 manager.go:1199] Started watching for new ooms in manager Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.254855 4557 manager.go:319] Starting recovery of all containers Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255403 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4b58f2a-a421-45cc-ba47-3b7519d9d074" volumeName="kubernetes.io/configmap/a4b58f2a-a421-45cc-ba47-3b7519d9d074-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255446 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bda5e742-65f0-4b52-a3d9-b6e3a222ba4d" volumeName="kubernetes.io/configmap/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d-serviceca" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255458 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3" volumeName="kubernetes.io/empty-dir/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-tmp" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255470 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bda5e742-65f0-4b52-a3d9-b6e3a222ba4d" volumeName="kubernetes.io/projected/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d-kube-api-access-gvvr8" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255479 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3" volumeName="kubernetes.io/projected/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-kube-api-access-2z4c7" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255489 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee7e4109-c1b4-4074-9c1e-cb40533b8df4" volumeName="kubernetes.io/configmap/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-config-volume" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255497 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="731f41d1-b212-4b32-b0e1-b4d4188ed782" volumeName="kubernetes.io/configmap/731f41d1-b212-4b32-b0e1-b4d4188ed782-env-overrides" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255505 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="731f41d1-b212-4b32-b0e1-b4d4188ed782" volumeName="kubernetes.io/projected/731f41d1-b212-4b32-b0e1-b4d4188ed782-kube-api-access-92vrw" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255519 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1bcd546-0077-45d1-b7bd-bad96f267ad6" volumeName="kubernetes.io/projected/a1bcd546-0077-45d1-b7bd-bad96f267ad6-kube-api-access-dfrqv" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255526 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee7e4109-c1b4-4074-9c1e-cb40533b8df4" volumeName="kubernetes.io/empty-dir/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-tmp-dir" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255537 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4b58f2a-a421-45cc-ba47-3b7519d9d074" volumeName="kubernetes.io/configmap/a4b58f2a-a421-45cc-ba47-3b7519d9d074-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255545 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d17094e-0f10-4e31-9afd-53daabac8d2d" volumeName="kubernetes.io/secret/1d17094e-0f10-4e31-9afd-53daabac8d2d-signing-key" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255556 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3723d5ac-70f9-42db-a027-7f84c5d622fa" volumeName="kubernetes.io/empty-dir/3723d5ac-70f9-42db-a027-7f84c5d622fa-data-volume" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255563 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="444ebadb-41a3-4961-bc14-e2bc23ab2e93" volumeName="kubernetes.io/projected/444ebadb-41a3-4961-bc14-e2bc23ab2e93-kube-api-access-wnrvl" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255591 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="450408b0-4d59-4e6d-a643-a2d1e15e459e" volumeName="kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-service-ca-bundle" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255615 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c87d997-baa8-434f-9ccb-97ade4abe46d" volumeName="kubernetes.io/secret/6c87d997-baa8-434f-9ccb-97ade4abe46d-cert" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255626 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="731f41d1-b212-4b32-b0e1-b4d4188ed782" volumeName="kubernetes.io/secret/731f41d1-b212-4b32-b0e1-b4d4188ed782-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255642 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5c66479-e6b2-40cb-acc4-5f61db6a778a" volumeName="kubernetes.io/projected/c5c66479-e6b2-40cb-acc4-5f61db6a778a-kube-api-access-vbmc7" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255654 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d17094e-0f10-4e31-9afd-53daabac8d2d" volumeName="kubernetes.io/configmap/1d17094e-0f10-4e31-9afd-53daabac8d2d-signing-cabundle" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255669 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3723d5ac-70f9-42db-a027-7f84c5d622fa" volumeName="kubernetes.io/projected/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-api-access-tbxw5" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255683 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="731f41d1-b212-4b32-b0e1-b4d4188ed782" volumeName="kubernetes.io/configmap/731f41d1-b212-4b32-b0e1-b4d4188ed782-ovnkube-script-lib" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255704 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d399188c-4d8b-4769-b212-bb2b2c485376" volumeName="kubernetes.io/configmap/d399188c-4d8b-4769-b212-bb2b2c485376-cni-binary-copy" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255720 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d399188c-4d8b-4769-b212-bb2b2c485376" volumeName="kubernetes.io/projected/d399188c-4d8b-4769-b212-bb2b2c485376-kube-api-access-lw4n9" seLinuxMountContext="" Nov 03 15:08:10.255708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255732 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3723d5ac-70f9-42db-a027-7f84c5d622fa" volumeName="kubernetes.io/configmap/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255753 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50838860-ed04-41fa-b347-136f03437c18" volumeName="kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255766 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1360ffe-4b52-4b77-8fbc-7a74c3b052a3" volumeName="kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255786 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4b58f2a-a421-45cc-ba47-3b7519d9d074" volumeName="kubernetes.io/configmap/a4b58f2a-a421-45cc-ba47-3b7519d9d074-cni-binary-copy" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255800 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c432b1d0-d7d8-49c4-a26f-716b10424bc9" volumeName="kubernetes.io/configmap/c432b1d0-d7d8-49c4-a26f-716b10424bc9-konnectivity-ca" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255822 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2670401-d2a2-49fd-817a-18e04f38d7de" volumeName="kubernetes.io/projected/d2670401-d2a2-49fd-817a-18e04f38d7de-kube-api-access-p7m7w" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255833 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d399188c-4d8b-4769-b212-bb2b2c485376" volumeName="kubernetes.io/configmap/d399188c-4d8b-4769-b212-bb2b2c485376-multus-daemon-config" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255848 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="450408b0-4d59-4e6d-a643-a2d1e15e459e" volumeName="kubernetes.io/projected/450408b0-4d59-4e6d-a643-a2d1e15e459e-kube-api-access-fhhfg" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255861 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c87d997-baa8-434f-9ccb-97ade4abe46d" volumeName="kubernetes.io/projected/6c87d997-baa8-434f-9ccb-97ade4abe46d-kube-api-access-jr9c8" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255872 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c432b1d0-d7d8-49c4-a26f-716b10424bc9" volumeName="kubernetes.io/secret/c432b1d0-d7d8-49c4-a26f-716b10424bc9-agent-certs" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255883 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3" volumeName="kubernetes.io/empty-dir/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-tuned" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255891 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee7e4109-c1b4-4074-9c1e-cb40533b8df4" volumeName="kubernetes.io/projected/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-kube-api-access-ctthh" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255901 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4b58f2a-a421-45cc-ba47-3b7519d9d074" volumeName="kubernetes.io/projected/a4b58f2a-a421-45cc-ba47-3b7519d9d074-kube-api-access-rtd4c" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255909 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="450408b0-4d59-4e6d-a643-a2d1e15e459e" volumeName="kubernetes.io/empty-dir/450408b0-4d59-4e6d-a643-a2d1e15e459e-archives-path" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255916 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1bcd546-0077-45d1-b7bd-bad96f267ad6" volumeName="kubernetes.io/configmap/a1bcd546-0077-45d1-b7bd-bad96f267ad6-iptables-alerter-script" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255927 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d17094e-0f10-4e31-9afd-53daabac8d2d" volumeName="kubernetes.io/projected/1d17094e-0f10-4e31-9afd-53daabac8d2d-kube-api-access-z9kjl" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255935 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="444ebadb-41a3-4961-bc14-e2bc23ab2e93" volumeName="kubernetes.io/empty-dir/444ebadb-41a3-4961-bc14-e2bc23ab2e93-tmp-dir" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255945 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="450408b0-4d59-4e6d-a643-a2d1e15e459e" volumeName="kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-trusted-ca-bundle" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255953 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a425029-f78c-4331-bb2a-00a94f6305d3" volumeName="kubernetes.io/projected/4a425029-f78c-4331-bb2a-00a94f6305d3-kube-api-access-5zt9k" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255961 4557 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="731f41d1-b212-4b32-b0e1-b4d4188ed782" volumeName="kubernetes.io/configmap/731f41d1-b212-4b32-b0e1-b4d4188ed782-ovnkube-config" seLinuxMountContext="" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255968 4557 reconstruct.go:97] "Volume reconstruction finished" Nov 03 15:08:10.257291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.255974 4557 reconciler.go:29] "Reconciler: start to sync state" Nov 03 15:08:10.276254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.276216 4557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 15:08:10.277752 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.277724 4557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 15:08:10.277752 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.277750 4557 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 15:08:10.277936 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.277775 4557 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 15:08:10.277936 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:10.277837 4557 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 15:08:10.284258 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.284229 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:08:10.284385 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.284344 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:08:10.296271 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.296244 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:08:10.296639 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.296613 4557 generic.go:356] "Generic (PLEG): container finished" podID="731f41d1-b212-4b32-b0e1-b4d4188ed782" containerID="a165e8f37384f6f0efa4f635a055af6815fb66e2e39fd0ab2a45edab5ef7b245" exitCode=1 Nov 03 15:08:10.298544 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.298522 4557 generic.go:356] "Generic (PLEG): container finished" podID="c495e56c3e03d09d80c3dcb0d6aed9eb" containerID="2c000505ce1e75e2cd4de3ecc4abb306f77debe887bab0f5ee3de41c7a43b924" exitCode=0 Nov 03 15:08:10.310235 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.310202 4557 generic.go:356] "Generic (PLEG): container finished" podID="a4b58f2a-a421-45cc-ba47-3b7519d9d074" containerID="6c33ae8b5a1798de3e82de952e48eb477f844ca5f9f9f330f1ad6d829f495c33" exitCode=0 Nov 03 15:08:10.310235 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.310231 4557 generic.go:356] "Generic (PLEG): container finished" podID="a4b58f2a-a421-45cc-ba47-3b7519d9d074" containerID="15bc0f58e067166b9fd2776a3ff9d5d34b9458a4d4235f3ebf641b4aa4e63b14" exitCode=0 Nov 03 15:08:10.310235 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.310240 4557 generic.go:356] "Generic (PLEG): container finished" podID="a4b58f2a-a421-45cc-ba47-3b7519d9d074" containerID="6279b122fa39faee7432f4c03150a2fcc4a3710e6affedaed44ec19e9e31300e" exitCode=0 Nov 03 15:08:10.310429 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.310250 4557 generic.go:356] "Generic (PLEG): container finished" podID="a4b58f2a-a421-45cc-ba47-3b7519d9d074" containerID="1442b279dad68058dbe4dfb180606f435a67474bcf4c897f1c2522519efc02f8" exitCode=0 Nov 03 15:08:10.310429 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.310287 4557 generic.go:356] "Generic (PLEG): container finished" podID="a4b58f2a-a421-45cc-ba47-3b7519d9d074" containerID="9e3328274ee8bbf49d26cfba5a2b904568d7486ebbaeb801ea70cfc3888cc02b" exitCode=0 Nov 03 15:08:10.310429 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.310297 4557 generic.go:356] "Generic (PLEG): container finished" podID="a4b58f2a-a421-45cc-ba47-3b7519d9d074" containerID="813bf54ab27f38f3fd68385dab613ef0c0dfe02c01129ab75b59d1ab8150afb3" exitCode=0 Nov 03 15:08:10.334405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.334338 4557 manager.go:324] Recovery completed Nov 03 15:08:10.347660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.347641 4557 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:08:10.348765 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.348745 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:08:10.348832 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.348786 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:08:10.348832 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.348797 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:08:10.350326 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.350260 4557 cpu_manager.go:226] "Starting" policy="none" Nov 03 15:08:10.350326 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.350273 4557 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 15:08:10.350326 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.350289 4557 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 15:08:10.350326 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.350310 4557 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:08:10.350564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.350440 4557 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 15:08:10.350564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.350457 4557 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 15:08:10.350564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.350483 4557 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 15:08:10.350564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.350492 4557 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 15:08:10.350564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.350499 4557 policy_none.go:50] "Start" Nov 03 15:08:10.350564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.350520 4557 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 15:08:10.350564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.350534 4557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 15:08:10.350942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.350690 4557 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 15:08:10.350942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.350698 4557 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 15:08:10.350942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.350703 4557 policy_none.go:44] "Start" Nov 03 15:08:10.351127 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.351110 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:08:10.351178 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.351136 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:08:10.351178 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.351146 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:08:10.354676 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:10.354655 4557 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-207.us-west-1.compute.internal\" not found" Nov 03 15:08:10.354780 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.354742 4557 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 15:08:10.361200 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.361182 4557 manager.go:341] "Starting Device Plugin manager" Nov 03 15:08:10.361326 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:10.361210 4557 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 15:08:10.361326 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.361219 4557 server.go:87] "Starting device plugin registration server" Nov 03 15:08:10.361483 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.361470 4557 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 15:08:10.361540 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.361484 4557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 15:08:10.361616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.361595 4557 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 15:08:10.361767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.361731 4557 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 15:08:10.361845 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.361774 4557 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 15:08:10.361845 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.361781 4557 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 15:08:10.361943 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.361840 4557 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T15:08:10.361764263Z","UUID":"9cfa0ff0-ae3d-484a-8783-c7c51afa4017","Handler":null,"Name":"","Endpoint":""} Nov 03 15:08:10.363153 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:10.363133 4557 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 15:08:10.363230 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:10.363171 4557 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-207.us-west-1.compute.internal\" not found" Nov 03 15:08:10.363999 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.363980 4557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 15:08:10.364082 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.364019 4557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 15:08:10.378353 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.377900 4557 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal"] Nov 03 15:08:10.378445 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.378434 4557 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:08:10.379222 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.379206 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:08:10.379296 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.379239 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:08:10.379296 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.379254 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:08:10.379609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.379598 4557 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:08:10.379711 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.379698 4557 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:08:10.380358 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.380338 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:08:10.380433 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.380368 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:08:10.380433 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.380380 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:08:10.380433 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.380345 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:08:10.380556 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.380434 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:08:10.380556 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.380449 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:08:10.380662 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:10.380604 4557 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-207.us-west-1.compute.internal\" not found" node="ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:10.380924 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.380865 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal" event={"ID":"abb209747bf3463921ac94593d5ab068","Type":"ContainerStarted","Data":"9695eaeda0651f87261d76e1f35e9ebd3b48cbfc715f475cd0cbf1427e5c7396"} Nov 03 15:08:10.381009 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.380931 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal" event={"ID":"abb209747bf3463921ac94593d5ab068","Type":"ContainerStarted","Data":"1fb72603852cc9ae8393bf638086357ec418a7b05a3b7b55940e48d760043fda"} Nov 03 15:08:10.381009 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.380992 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" event={"ID":"c495e56c3e03d09d80c3dcb0d6aed9eb","Type":"ContainerStarted","Data":"19c3f6381f95dcae4f0fe525b3811c22d67773beacfd8bde7e3850cb76f307d4"} Nov 03 15:08:10.381108 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.381011 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" event={"ID":"c495e56c3e03d09d80c3dcb0d6aed9eb","Type":"ContainerDied","Data":"2c000505ce1e75e2cd4de3ecc4abb306f77debe887bab0f5ee3de41c7a43b924"} Nov 03 15:08:10.381108 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.381023 4557 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:08:10.381108 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.381030 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" event={"ID":"c495e56c3e03d09d80c3dcb0d6aed9eb","Type":"ContainerStarted","Data":"97b5a2edac4885dfb5539d31ff3f862082f6ae68c9dd2287ed3355d21b1f9b34"} Nov 03 15:08:10.381787 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.381772 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:08:10.381877 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.381800 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:08:10.381877 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.381810 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:08:10.381957 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:10.381941 4557 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-207.us-west-1.compute.internal\" not found" node="ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:10.404358 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.404332 4557 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:08:10.453811 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.453784 4557 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 15:08:10.461693 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.461667 4557 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 15:08:10.462798 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.462782 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 15:08:10.462864 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.462815 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 15:08:10.462864 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.462825 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 15:08:10.462864 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.462856 4557 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:10.502060 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.502021 4557 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:08:10.503973 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.503955 4557 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:08:10.504533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.504517 4557 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 15:08:10.554694 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.554654 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c495e56c3e03d09d80c3dcb0d6aed9eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal\" (UID: \"c495e56c3e03d09d80c3dcb0d6aed9eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:10.554694 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.554678 4557 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:10.554939 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.554737 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c495e56c3e03d09d80c3dcb0d6aed9eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal\" (UID: \"c495e56c3e03d09d80c3dcb0d6aed9eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:10.554939 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.554781 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/abb209747bf3463921ac94593d5ab068-config\") pod \"kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal\" (UID: \"abb209747bf3463921ac94593d5ab068\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:10.599667 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.599556 4557 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:10.599667 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.599670 4557 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:10.600284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.600268 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeNotReady" Nov 03 15:08:10.600340 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.600305 4557 setters.go:546] "Node became not ready" node="ip-10-0-139-207.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T15:08:10Z","lastTransitionTime":"2026-11-03T15:08:10Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 15:08:10.620661 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.620630 4557 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 15:08:10.620811 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:10.620756 4557 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:10.620811 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.620777 4557 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:10.655353 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.655323 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c495e56c3e03d09d80c3dcb0d6aed9eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal\" (UID: \"c495e56c3e03d09d80c3dcb0d6aed9eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:10.655507 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.655354 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c495e56c3e03d09d80c3dcb0d6aed9eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal\" (UID: \"c495e56c3e03d09d80c3dcb0d6aed9eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:10.655507 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.655377 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/abb209747bf3463921ac94593d5ab068-config\") pod \"kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal\" (UID: \"abb209747bf3463921ac94593d5ab068\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:10.655507 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.655432 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c495e56c3e03d09d80c3dcb0d6aed9eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal\" (UID: \"c495e56c3e03d09d80c3dcb0d6aed9eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:10.655507 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.655435 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c495e56c3e03d09d80c3dcb0d6aed9eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal\" (UID: \"c495e56c3e03d09d80c3dcb0d6aed9eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:10.655507 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.655503 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/abb209747bf3463921ac94593d5ab068-config\") pod \"kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal\" (UID: \"abb209747bf3463921ac94593d5ab068\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:10.673549 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.673517 4557 nodeinfomanager.go:471] existing CSINode "ip-10-0-139-207.us-west-1.compute.internal" is owned by different node (oldNodeID="7f12a31b-0735-4ef8-b584-2a1103f955da", newNodeID=""), cleaning up... Nov 03 15:08:10.679684 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:10.679659 4557 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 15:08:10.679785 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:10.679705 4557 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:11.240124 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.240083 4557 apiserver.go:52] "Watching apiserver" Nov 03 15:08:11.320704 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.320681 4557 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:11.320838 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.320780 4557 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:11.369646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.369619 4557 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 15:08:11.370221 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.370200 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-7vw5n","openshift-dns/dns-default-kpr4x","openshift-image-registry/node-ca-kr9hf","openshift-multus/multus-additional-cni-plugins-7k755","openshift-cluster-node-tuning-operator/tuned-6j7m4","openshift-insights/insights-runtime-extractor-x575x","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal","openshift-ingress-canary/ingress-canary-hmch9","openshift-service-ca/service-ca-5fffc97bdf-x79q9","kube-system/konnectivity-agent-mh4np","openshift-multus/multus-b5nrd","openshift-network-diagnostics/network-check-target-lws9f","openshift-ovn-kubernetes/ovnkube-node-mmvs8","openshift-dns/node-resolver-rznqb","kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal","openshift-insights/periodic-gathering-hwzmv-gkbt4","openshift-kube-storage-version-migrator/migrator-66df5d496b-x8qvd","openshift-network-operator/iptables-alerter-644l4","openshift-multus/network-metrics-daemon-w4gnx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v"] Nov 03 15:08:11.372655 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.372630 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:08:11.375665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.375644 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kpr4x" Nov 03 15:08:11.376413 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.376396 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:11.380345 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.380325 4557 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 15:08:11.380419 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:11.380382 4557 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:11.389376 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.389354 4557 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 15:08:11.389451 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:11.389409 4557 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-207.us-west-1.compute.internal" Nov 03 15:08:11.426648 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.426615 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 15:08:11.427279 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.427244 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 15:08:11.427406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.427380 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 15:08:11.427561 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.427545 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 15:08:11.427723 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.427694 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4wckw\"" Nov 03 15:08:11.427815 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.427800 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 15:08:11.427867 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.427813 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 15:08:11.427867 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.427848 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 15:08:11.427963 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.427886 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 15:08:11.427963 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.427905 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 15:08:11.427963 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.427954 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 15:08:11.428097 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.427997 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fsh2r\"" Nov 03 15:08:11.428097 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.428058 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 15:08:11.428189 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.428114 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 15:08:11.428258 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.428241 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 15:08:11.428306 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.428276 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 15:08:11.428353 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.427547 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 15:08:11.428429 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.428409 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 15:08:11.428554 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.428537 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 15:08:11.428634 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.428609 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 15:08:11.428734 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.428715 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 15:08:11.429090 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.429071 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 15:08:11.429165 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.429137 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 15:08:11.429214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.429193 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 15:08:11.429261 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.429228 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 15:08:11.429429 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.429409 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 15:08:11.429485 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.429473 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 15:08:11.429883 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.429866 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cblvv\"" Nov 03 15:08:11.430142 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.430123 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qsch2\"" Nov 03 15:08:11.430228 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.430129 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 15:08:11.430228 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.430213 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cjtnz\"" Nov 03 15:08:11.430312 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.430295 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kdt56\"" Nov 03 15:08:11.430429 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.430414 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dfz9p\"" Nov 03 15:08:11.430636 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.430620 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 15:08:11.430815 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.430795 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-975kv\"" Nov 03 15:08:11.430878 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.430841 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 15:08:11.430976 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.430954 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z77jb\"" Nov 03 15:08:11.431036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.430999 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 15:08:11.431036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.431019 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6cpt6\"" Nov 03 15:08:11.431333 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.431314 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 15:08:11.431333 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.431312 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 15:08:11.431508 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.431350 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qbvnh\"" Nov 03 15:08:11.431508 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.431416 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jpxkd\"" Nov 03 15:08:11.454538 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.454513 4557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 15:08:11.460717 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.460695 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zt9k\" (UniqueName: \"kubernetes.io/projected/4a425029-f78c-4331-bb2a-00a94f6305d3-kube-api-access-5zt9k\") pod \"migrator-66df5d496b-x8qvd\" (UID: \"4a425029-f78c-4331-bb2a-00a94f6305d3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x8qvd" Nov 03 15:08:11.460829 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.460733 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-device-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.460829 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.460760 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-run-systemd\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.460829 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.460785 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/444ebadb-41a3-4961-bc14-e2bc23ab2e93-tmp-dir\") pod \"node-resolver-rznqb\" (UID: \"444ebadb-41a3-4961-bc14-e2bc23ab2e93\") " pod="openshift-dns/node-resolver-rznqb" Nov 03 15:08:11.460958 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.460936 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/444ebadb-41a3-4961-bc14-e2bc23ab2e93-tmp-dir\") pod \"node-resolver-rznqb\" (UID: \"444ebadb-41a3-4961-bc14-e2bc23ab2e93\") " pod="openshift-dns/node-resolver-rznqb" Nov 03 15:08:11.461007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.460986 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3723d5ac-70f9-42db-a027-7f84c5d622fa-data-volume\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:11.461038 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461025 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:08:11.461067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461055 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z4c7\" (UniqueName: \"kubernetes.io/projected/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-kube-api-access-2z4c7\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.461103 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461081 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d-serviceca\") pod \"node-ca-kr9hf\" (UID: \"bda5e742-65f0-4b52-a3d9-b6e3a222ba4d\") " pod="openshift-image-registry/node-ca-kr9hf" Nov 03 15:08:11.461103 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461086 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3723d5ac-70f9-42db-a027-7f84c5d622fa-data-volume\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:11.461174 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461107 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs\") pod \"network-metrics-daemon-w4gnx\" (UID: \"d2670401-d2a2-49fd-817a-18e04f38d7de\") " pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:08:11.461174 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461163 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-etc-openvswitch\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.461260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461247 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92vrw\" (UniqueName: \"kubernetes.io/projected/731f41d1-b212-4b32-b0e1-b4d4188ed782-kube-api-access-92vrw\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.461299 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461273 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/444ebadb-41a3-4961-bc14-e2bc23ab2e93-hosts-file\") pod \"node-resolver-rznqb\" (UID: \"444ebadb-41a3-4961-bc14-e2bc23ab2e93\") " pod="openshift-dns/node-resolver-rznqb" Nov 03 15:08:11.461347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461297 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4b58f2a-a421-45cc-ba47-3b7519d9d074-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:08:11.461400 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461354 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvvr8\" (UniqueName: \"kubernetes.io/projected/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d-kube-api-access-gvvr8\") pod \"node-ca-kr9hf\" (UID: \"bda5e742-65f0-4b52-a3d9-b6e3a222ba4d\") " pod="openshift-image-registry/node-ca-kr9hf" Nov 03 15:08:11.461400 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461369 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d-serviceca\") pod \"node-ca-kr9hf\" (UID: \"bda5e742-65f0-4b52-a3d9-b6e3a222ba4d\") " pod="openshift-image-registry/node-ca-kr9hf" Nov 03 15:08:11.461400 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461389 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-sysctl-conf\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.461509 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461422 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d17094e-0f10-4e31-9afd-53daabac8d2d-signing-key\") pod \"service-ca-5fffc97bdf-x79q9\" (UID: \"1d17094e-0f10-4e31-9afd-53daabac8d2d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" Nov 03 15:08:11.461509 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461426 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-original-pull-secret\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:08:11.461509 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461477 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-hostroot\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.461645 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461511 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4b58f2a-a421-45cc-ba47-3b7519d9d074-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:08:11.461645 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461519 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-run-multus-certs\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.461645 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461595 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtd4c\" (UniqueName: \"kubernetes.io/projected/a4b58f2a-a421-45cc-ba47-3b7519d9d074-kube-api-access-rtd4c\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:08:11.461645 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461620 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-run\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.461831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461647 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-tuned\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.461831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461664 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-tmp\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.461831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461681 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-multus-socket-dir-parent\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.461831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461701 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c87d997-baa8-434f-9ccb-97ade4abe46d-cert\") pod \"ingress-canary-hmch9\" (UID: \"6c87d997-baa8-434f-9ccb-97ade4abe46d\") " pod="openshift-ingress-canary/ingress-canary-hmch9" Nov 03 15:08:11.461831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461723 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4b58f2a-a421-45cc-ba47-3b7519d9d074-cni-binary-copy\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:08:11.461831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461801 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-tuned\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.461831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461801 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-tmp\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.461831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461822 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-modprobe-d\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.462102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461896 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-registration-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.462102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461915 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbmc7\" (UniqueName: \"kubernetes.io/projected/c5c66479-e6b2-40cb-acc4-5f61db6a778a-kube-api-access-vbmc7\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.462102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461930 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4b58f2a-a421-45cc-ba47-3b7519d9d074-cni-binary-copy\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:08:11.462102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.461996 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-run-ovn\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.462102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462018 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4b58f2a-a421-45cc-ba47-3b7519d9d074-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:08:11.462102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462035 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-kubernetes\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.462102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462066 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-multus-cni-dir\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.462378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462112 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-etc-kubernetes\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.462378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462145 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.462378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462173 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/731f41d1-b212-4b32-b0e1-b4d4188ed782-ovn-node-metrics-cert\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.462378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462193 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-tmp-dir\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:08:11.462378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462260 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-tmp-dir\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:08:11.462378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462309 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbxw5\" (UniqueName: \"kubernetes.io/projected/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-api-access-tbxw5\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:11.462378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462339 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-trusted-ca-bundle\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:11.462378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462341 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/731f41d1-b212-4b32-b0e1-b4d4188ed782-ovn-node-metrics-cert\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.462378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462363 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhhfg\" (UniqueName: \"kubernetes.io/projected/450408b0-4d59-4e6d-a643-a2d1e15e459e-kube-api-access-fhhfg\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:11.462712 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462496 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-sysctl-d\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.462712 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462530 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-kubelet-config\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:08:11.462712 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462554 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d-host\") pod \"node-ca-kr9hf\" (UID: \"bda5e742-65f0-4b52-a3d9-b6e3a222ba4d\") " pod="openshift-image-registry/node-ca-kr9hf" Nov 03 15:08:11.462712 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462600 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-var-lib-cni-bin\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.462712 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462629 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-kubelet\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.462712 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462658 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-node-log\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.462712 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462687 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfrqv\" (UniqueName: \"kubernetes.io/projected/a1bcd546-0077-45d1-b7bd-bad96f267ad6-kube-api-access-dfrqv\") pod \"iptables-alerter-644l4\" (UID: \"a1bcd546-0077-45d1-b7bd-bad96f267ad6\") " pod="openshift-network-operator/iptables-alerter-644l4" Nov 03 15:08:11.462928 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462716 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-socket-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.462928 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462807 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/731f41d1-b212-4b32-b0e1-b4d4188ed782-ovnkube-script-lib\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.462928 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462845 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4b58f2a-a421-45cc-ba47-3b7519d9d074-system-cni-dir\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:08:11.462928 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462861 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4b58f2a-a421-45cc-ba47-3b7519d9d074-cnibin\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:08:11.462928 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462876 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:08:11.463147 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462940 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d17094e-0f10-4e31-9afd-53daabac8d2d-signing-cabundle\") pod \"service-ca-5fffc97bdf-x79q9\" (UID: \"1d17094e-0f10-4e31-9afd-53daabac8d2d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" Nov 03 15:08:11.463147 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.462963 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-cnibin\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.463147 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463000 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d399188c-4d8b-4769-b212-bb2b2c485376-cni-binary-copy\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.463147 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463066 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-run-netns\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.463147 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463117 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw4n9\" (UniqueName: \"kubernetes.io/projected/d399188c-4d8b-4769-b212-bb2b2c485376-kube-api-access-lw4n9\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.463360 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463155 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr9c8\" (UniqueName: \"kubernetes.io/projected/6c87d997-baa8-434f-9ccb-97ade4abe46d-kube-api-access-jr9c8\") pod \"ingress-canary-hmch9\" (UID: \"6c87d997-baa8-434f-9ccb-97ade4abe46d\") " pod="openshift-ingress-canary/ingress-canary-hmch9" Nov 03 15:08:11.463360 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463177 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/731f41d1-b212-4b32-b0e1-b4d4188ed782-ovnkube-script-lib\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.463360 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463191 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4b58f2a-a421-45cc-ba47-3b7519d9d074-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:08:11.463360 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463299 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d399188c-4d8b-4769-b212-bb2b2c485376-cni-binary-copy\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.463492 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463361 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4b58f2a-a421-45cc-ba47-3b7519d9d074-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:08:11.463492 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463372 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.463492 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463396 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-systemd-units\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.463492 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463434 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-var-lib-openvswitch\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.463492 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463456 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3723d5ac-70f9-42db-a027-7f84c5d622fa-crio-socket\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:11.463492 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463490 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7m7w\" (UniqueName: \"kubernetes.io/projected/d2670401-d2a2-49fd-817a-18e04f38d7de-kube-api-access-p7m7w\") pod \"network-metrics-daemon-w4gnx\" (UID: \"d2670401-d2a2-49fd-817a-18e04f38d7de\") " pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:08:11.463850 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463512 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1bcd546-0077-45d1-b7bd-bad96f267ad6-host-slash\") pod \"iptables-alerter-644l4\" (UID: \"a1bcd546-0077-45d1-b7bd-bad96f267ad6\") " pod="openshift-network-operator/iptables-alerter-644l4" Nov 03 15:08:11.463850 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463557 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-lib-modules\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.463850 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463576 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c432b1d0-d7d8-49c4-a26f-716b10424bc9-agent-certs\") pod \"konnectivity-agent-mh4np\" (UID: \"c432b1d0-d7d8-49c4-a26f-716b10424bc9\") " pod="kube-system/konnectivity-agent-mh4np" Nov 03 15:08:11.463850 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463680 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c432b1d0-d7d8-49c4-a26f-716b10424bc9-konnectivity-ca\") pod \"konnectivity-agent-mh4np\" (UID: \"c432b1d0-d7d8-49c4-a26f-716b10424bc9\") " pod="kube-system/konnectivity-agent-mh4np" Nov 03 15:08:11.463850 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463708 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-multus-conf-dir\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.463850 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463766 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1bcd546-0077-45d1-b7bd-bad96f267ad6-iptables-alerter-script\") pod \"iptables-alerter-644l4\" (UID: \"a1bcd546-0077-45d1-b7bd-bad96f267ad6\") " pod="openshift-network-operator/iptables-alerter-644l4" Nov 03 15:08:11.463850 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463806 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-systemd\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.463850 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463823 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c432b1d0-d7d8-49c4-a26f-716b10424bc9-agent-certs\") pod \"konnectivity-agent-mh4np\" (UID: \"c432b1d0-d7d8-49c4-a26f-716b10424bc9\") " pod="kube-system/konnectivity-agent-mh4np" Nov 03 15:08:11.463850 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463829 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-var-lib-kubelet\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.464188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463879 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9kjl\" (UniqueName: \"kubernetes.io/projected/1d17094e-0f10-4e31-9afd-53daabac8d2d-kube-api-access-z9kjl\") pod \"service-ca-5fffc97bdf-x79q9\" (UID: \"1d17094e-0f10-4e31-9afd-53daabac8d2d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" Nov 03 15:08:11.464188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463953 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c432b1d0-d7d8-49c4-a26f-716b10424bc9-konnectivity-ca\") pod \"konnectivity-agent-mh4np\" (UID: \"c432b1d0-d7d8-49c4-a26f-716b10424bc9\") " pod="kube-system/konnectivity-agent-mh4np" Nov 03 15:08:11.464188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.463965 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-sys-fs\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.464188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464007 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-run-k8s-cni-cncf-io\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.464188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464054 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-run-netns\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.464188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464065 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1bcd546-0077-45d1-b7bd-bad96f267ad6-iptables-alerter-script\") pod \"iptables-alerter-644l4\" (UID: \"a1bcd546-0077-45d1-b7bd-bad96f267ad6\") " pod="openshift-network-operator/iptables-alerter-644l4" Nov 03 15:08:11.464188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464095 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-run-openvswitch\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.464188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464136 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-dbus\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:08:11.464188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464156 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-sysconfig\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.464188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464187 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-log-socket\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.464499 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464204 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/731f41d1-b212-4b32-b0e1-b4d4188ed782-ovnkube-config\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.464499 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464218 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctthh\" (UniqueName: \"kubernetes.io/projected/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-kube-api-access-ctthh\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:08:11.464499 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464257 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:11.464499 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464287 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjgmv\" (UniqueName: \"kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv\") pod \"network-check-target-lws9f\" (UID: \"50838860-ed04-41fa-b347-136f03437c18\") " pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:08:11.464499 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464317 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-host\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.464499 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464334 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-service-ca-bundle\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:11.464499 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464369 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-system-cni-dir\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.464499 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464389 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-slash\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.464499 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464407 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-run-ovn-kubernetes\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.464499 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464427 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/731f41d1-b212-4b32-b0e1-b4d4188ed782-ovnkube-config\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.464499 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464442 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-cni-bin\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.464499 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464470 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/731f41d1-b212-4b32-b0e1-b4d4188ed782-env-overrides\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.464499 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464489 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/450408b0-4d59-4e6d-a643-a2d1e15e459e-archives-path\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:11.465048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464569 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/450408b0-4d59-4e6d-a643-a2d1e15e459e-archives-path\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:11.465048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464625 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-sys\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.465048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464649 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/731f41d1-b212-4b32-b0e1-b4d4188ed782-env-overrides\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.465048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464680 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-etc-selinux\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.465048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464721 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-var-lib-cni-multus\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.465048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464748 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-var-lib-kubelet\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.465048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464774 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-config-volume\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:08:11.465048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464820 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:11.465048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464854 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d399188c-4d8b-4769-b212-bb2b2c485376-multus-daemon-config\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.465048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464879 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-cni-netd\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.465048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.464916 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnrvl\" (UniqueName: \"kubernetes.io/projected/444ebadb-41a3-4961-bc14-e2bc23ab2e93-kube-api-access-wnrvl\") pod \"node-resolver-rznqb\" (UID: \"444ebadb-41a3-4961-bc14-e2bc23ab2e93\") " pod="openshift-dns/node-resolver-rznqb" Nov 03 15:08:11.465372 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.465089 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d399188c-4d8b-4769-b212-bb2b2c485376-multus-daemon-config\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.472903 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.472885 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 15:08:11.472987 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.472971 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 15:08:11.473214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.473192 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 15:08:11.473214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.473200 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 15:08:11.473426 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.473410 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 15:08:11.473480 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.473411 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 15:08:11.473480 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.473469 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 15:08:11.473709 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.473697 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 15:08:11.473779 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.473765 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 15:08:11.473829 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.473786 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 15:08:11.473829 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.473802 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 15:08:11.473912 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.473893 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 15:08:11.474545 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.474532 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-service-ca-bundle\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:11.474645 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.474627 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:11.475007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.474988 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-config-volume\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:08:11.475740 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.475699 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:11.477771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.477753 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 15:08:11.483353 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.483338 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d17094e-0f10-4e31-9afd-53daabac8d2d-signing-cabundle\") pod \"service-ca-5fffc97bdf-x79q9\" (UID: \"1d17094e-0f10-4e31-9afd-53daabac8d2d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" Nov 03 15:08:11.483669 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.483650 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-trusted-ca-bundle\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:11.522994 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.522924 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92vrw\" (UniqueName: \"kubernetes.io/projected/731f41d1-b212-4b32-b0e1-b4d4188ed782-kube-api-access-92vrw\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.526033 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.526008 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z4c7\" (UniqueName: \"kubernetes.io/projected/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-kube-api-access-2z4c7\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.527497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.527471 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfrqv\" (UniqueName: \"kubernetes.io/projected/a1bcd546-0077-45d1-b7bd-bad96f267ad6-kube-api-access-dfrqv\") pod \"iptables-alerter-644l4\" (UID: \"a1bcd546-0077-45d1-b7bd-bad96f267ad6\") " pod="openshift-network-operator/iptables-alerter-644l4" Nov 03 15:08:11.527739 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.527721 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 15:08:11.527810 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.527758 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 15:08:11.528108 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.528079 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 15:08:11.528188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.528104 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvvr8\" (UniqueName: \"kubernetes.io/projected/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d-kube-api-access-gvvr8\") pod \"node-ca-kr9hf\" (UID: \"bda5e742-65f0-4b52-a3d9-b6e3a222ba4d\") " pod="openshift-image-registry/node-ca-kr9hf" Nov 03 15:08:11.528188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.528109 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr9c8\" (UniqueName: \"kubernetes.io/projected/6c87d997-baa8-434f-9ccb-97ade4abe46d-kube-api-access-jr9c8\") pod \"ingress-canary-hmch9\" (UID: \"6c87d997-baa8-434f-9ccb-97ade4abe46d\") " pod="openshift-ingress-canary/ingress-canary-hmch9" Nov 03 15:08:11.528188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.528131 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ncdvq\"" Nov 03 15:08:11.528188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.528083 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zncwb\"" Nov 03 15:08:11.528188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.528143 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pzzbk\"" Nov 03 15:08:11.528391 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.528246 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ltlwc\"" Nov 03 15:08:11.528391 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.528248 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g9729\"" Nov 03 15:08:11.528391 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.528308 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7mf6v\"" Nov 03 15:08:11.528391 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.528364 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 15:08:11.528502 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.528432 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbxw5\" (UniqueName: \"kubernetes.io/projected/3723d5ac-70f9-42db-a027-7f84c5d622fa-kube-api-access-tbxw5\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:11.529736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.529718 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zt9k\" (UniqueName: \"kubernetes.io/projected/4a425029-f78c-4331-bb2a-00a94f6305d3-kube-api-access-5zt9k\") pod \"migrator-66df5d496b-x8qvd\" (UID: \"4a425029-f78c-4331-bb2a-00a94f6305d3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x8qvd" Nov 03 15:08:11.530643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.530623 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtd4c\" (UniqueName: \"kubernetes.io/projected/a4b58f2a-a421-45cc-ba47-3b7519d9d074-kube-api-access-rtd4c\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:08:11.531890 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.531872 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d17094e-0f10-4e31-9afd-53daabac8d2d-signing-key\") pod \"service-ca-5fffc97bdf-x79q9\" (UID: \"1d17094e-0f10-4e31-9afd-53daabac8d2d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" Nov 03 15:08:11.532040 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.532024 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c87d997-baa8-434f-9ccb-97ade4abe46d-cert\") pod \"ingress-canary-hmch9\" (UID: \"6c87d997-baa8-434f-9ccb-97ade4abe46d\") " pod="openshift-ingress-canary/ingress-canary-hmch9" Nov 03 15:08:11.532321 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.532297 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhhfg\" (UniqueName: \"kubernetes.io/projected/450408b0-4d59-4e6d-a643-a2d1e15e459e-kube-api-access-fhhfg\") pod \"periodic-gathering-hwzmv-gkbt4\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:08:11.532379 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.532337 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbmc7\" (UniqueName: \"kubernetes.io/projected/c5c66479-e6b2-40cb-acc4-5f61db6a778a-kube-api-access-vbmc7\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.537645 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.537630 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjgmv\" (UniqueName: \"kubernetes.io/projected/50838860-ed04-41fa-b347-136f03437c18-kube-api-access-mjgmv\") pod \"network-check-target-lws9f\" (UID: \"50838860-ed04-41fa-b347-136f03437c18\") " pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:08:11.539206 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.539187 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctthh\" (UniqueName: \"kubernetes.io/projected/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-kube-api-access-ctthh\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:08:11.539297 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.539221 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw4n9\" (UniqueName: \"kubernetes.io/projected/d399188c-4d8b-4769-b212-bb2b2c485376-kube-api-access-lw4n9\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.539527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.539511 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnrvl\" (UniqueName: \"kubernetes.io/projected/444ebadb-41a3-4961-bc14-e2bc23ab2e93-kube-api-access-wnrvl\") pod \"node-resolver-rznqb\" (UID: \"444ebadb-41a3-4961-bc14-e2bc23ab2e93\") " pod="openshift-dns/node-resolver-rznqb" Nov 03 15:08:11.543570 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.543552 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9kjl\" (UniqueName: \"kubernetes.io/projected/1d17094e-0f10-4e31-9afd-53daabac8d2d-kube-api-access-z9kjl\") pod \"service-ca-5fffc97bdf-x79q9\" (UID: \"1d17094e-0f10-4e31-9afd-53daabac8d2d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x79q9" Nov 03 15:08:11.543673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.543626 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7m7w\" (UniqueName: \"kubernetes.io/projected/d2670401-d2a2-49fd-817a-18e04f38d7de-kube-api-access-p7m7w\") pod \"network-metrics-daemon-w4gnx\" (UID: \"d2670401-d2a2-49fd-817a-18e04f38d7de\") " pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:08:11.566143 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566116 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-sysctl-d\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.566143 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566145 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-kubelet-config\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:08:11.566318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566160 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d-host\") pod \"node-ca-kr9hf\" (UID: \"bda5e742-65f0-4b52-a3d9-b6e3a222ba4d\") " pod="openshift-image-registry/node-ca-kr9hf" Nov 03 15:08:11.566318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566177 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-var-lib-cni-bin\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.566318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566194 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-kubelet\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.566318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566229 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-node-log\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.566318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566252 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bda5e742-65f0-4b52-a3d9-b6e3a222ba4d-host\") pod \"node-ca-kr9hf\" (UID: \"bda5e742-65f0-4b52-a3d9-b6e3a222ba4d\") " pod="openshift-image-registry/node-ca-kr9hf" Nov 03 15:08:11.566318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566254 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-socket-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.566318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566276 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-var-lib-cni-bin\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.566318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566285 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-node-log\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.566318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566286 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-kubelet\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.566318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566301 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-sysctl-d\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.566318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566252 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-kubelet-config\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:08:11.566318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566305 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4b58f2a-a421-45cc-ba47-3b7519d9d074-system-cni-dir\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:08:11.566318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566303 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-socket-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566339 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4b58f2a-a421-45cc-ba47-3b7519d9d074-system-cni-dir\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566351 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4b58f2a-a421-45cc-ba47-3b7519d9d074-cnibin\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566379 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566409 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-cnibin\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566433 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-run-netns\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566379 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4b58f2a-a421-45cc-ba47-3b7519d9d074-cnibin\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566472 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566480 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-cnibin\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566496 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-systemd-units\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566504 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-run-netns\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566523 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-var-lib-openvswitch\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566519 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566546 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-systemd-units\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566552 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3723d5ac-70f9-42db-a027-7f84c5d622fa-crio-socket\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566566 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-var-lib-openvswitch\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566599 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1bcd546-0077-45d1-b7bd-bad96f267ad6-host-slash\") pod \"iptables-alerter-644l4\" (UID: \"a1bcd546-0077-45d1-b7bd-bad96f267ad6\") " pod="openshift-network-operator/iptables-alerter-644l4" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566627 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-lib-modules\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566634 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3723d5ac-70f9-42db-a027-7f84c5d622fa-crio-socket\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566651 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-multus-conf-dir\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566656 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1bcd546-0077-45d1-b7bd-bad96f267ad6-host-slash\") pod \"iptables-alerter-644l4\" (UID: \"a1bcd546-0077-45d1-b7bd-bad96f267ad6\") " pod="openshift-network-operator/iptables-alerter-644l4" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566682 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-systemd\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566707 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-lib-modules\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566695 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-multus-conf-dir\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566707 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-var-lib-kubelet\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566750 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-systemd\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566755 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-var-lib-kubelet\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566760 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-sys-fs\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566766 4557 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566792 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-run-k8s-cni-cncf-io\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566806 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-sys-fs\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566816 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-run-netns\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566840 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-run-openvswitch\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566842 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-run-k8s-cni-cncf-io\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566863 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-dbus\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566872 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-run-netns\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566881 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-run-openvswitch\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566902 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-sysconfig\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566925 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-log-socket\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566951 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-host\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.566954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566971 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-system-cni-dir\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566985 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-sysconfig\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566984 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-log-socket\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566991 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-slash\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567020 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-host\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.566990 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a1360ffe-4b52-4b77-8fbc-7a74c3b052a3-dbus\") pod \"global-pull-secret-syncer-7vw5n\" (UID: \"a1360ffe-4b52-4b77-8fbc-7a74c3b052a3\") " pod="kube-system/global-pull-secret-syncer-7vw5n" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567031 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-run-ovn-kubernetes\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567057 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-cni-bin\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567023 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-slash\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567048 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-system-cni-dir\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567089 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-run-ovn-kubernetes\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567100 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-cni-bin\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567104 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-sys\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567134 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-etc-selinux\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567152 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-sys\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567161 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-var-lib-cni-multus\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567185 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-var-lib-kubelet\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567213 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567186 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-etc-selinux\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567240 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-cni-netd\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567262 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-var-lib-kubelet\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567264 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-device-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567282 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-var-lib-cni-multus\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567292 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-cni-netd\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567300 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-run-systemd\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567313 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-device-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567322 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-run-systemd\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567328 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs\") pod \"network-metrics-daemon-w4gnx\" (UID: \"d2670401-d2a2-49fd-817a-18e04f38d7de\") " pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567358 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-etc-openvswitch\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567374 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/444ebadb-41a3-4961-bc14-e2bc23ab2e93-hosts-file\") pod \"node-resolver-rznqb\" (UID: \"444ebadb-41a3-4961-bc14-e2bc23ab2e93\") " pod="openshift-dns/node-resolver-rznqb" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567388 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-sysctl-conf\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567404 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-hostroot\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567426 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-etc-openvswitch\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567426 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-run-multus-certs\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567444 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/444ebadb-41a3-4961-bc14-e2bc23ab2e93-hosts-file\") pod \"node-resolver-rznqb\" (UID: \"444ebadb-41a3-4961-bc14-e2bc23ab2e93\") " pod="openshift-dns/node-resolver-rznqb" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567454 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-host-run-multus-certs\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:11.567330 4557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567465 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-run\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567470 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-hostroot\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567499 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-multus-socket-dir-parent\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567526 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-run\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567526 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-modprobe-d\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567528 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-sysctl-conf\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:11.567555 4557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls podName:3723d5ac-70f9-42db-a027-7f84c5d622fa nodeName:}" failed. No retries permitted until 2026-11-03 15:08:12.067534396 +0000 UTC m=+1.910655943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x575x" (UID: "3723d5ac-70f9-42db-a027-7f84c5d622fa") : secret "insights-runtime-extractor-tls" not found Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567569 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-multus-socket-dir-parent\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567614 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-registration-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567631 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-modprobe-d\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567639 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-run-ovn\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567659 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c5c66479-e6b2-40cb-acc4-5f61db6a778a-registration-dir\") pod \"aws-ebs-csi-driver-node-m986v\" (UID: \"c5c66479-e6b2-40cb-acc4-5f61db6a778a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m986v" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567664 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4b58f2a-a421-45cc-ba47-3b7519d9d074-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567695 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-kubernetes\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567703 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-run-ovn\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567723 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-multus-cni-dir\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567737 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4b58f2a-a421-45cc-ba47-3b7519d9d074-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7k755\" (UID: \"a4b58f2a-a421-45cc-ba47-3b7519d9d074\") " pod="openshift-multus/multus-additional-cni-plugins-7k755" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567750 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-etc-kubernetes\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567754 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3-etc-kubernetes\") pod \"tuned-6j7m4\" (UID: \"dbb8d4e4-6660-4a11-afe1-dc2aacfc49d3\") " pod="openshift-cluster-node-tuning-operator/tuned-6j7m4" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567781 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567798 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-etc-kubernetes\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567796 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d399188c-4d8b-4769-b212-bb2b2c485376-multus-cni-dir\") pod \"multus-b5nrd\" (UID: \"d399188c-4d8b-4769-b212-bb2b2c485376\") " pod="openshift-multus/multus-b5nrd" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.567833 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/731f41d1-b212-4b32-b0e1-b4d4188ed782-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mmvs8\" (UID: \"731f41d1-b212-4b32-b0e1-b4d4188ed782\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:11.568863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.568901 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee7e4109-c1b4-4074-9c1e-cb40533b8df4-metrics-tls\") pod \"dns-default-kpr4x\" (UID: \"ee7e4109-c1b4-4074-9c1e-cb40533b8df4\") " pod="openshift-dns/dns-default-kpr4x" Nov 03 15:08:11.570940 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.569814 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2670401-d2a2-49fd-817a-18e04f38d7de-metrics-certs\") pod \"network-metrics-daemon-w4gnx\" (UID: \"d2670401-d2a2-49fd-817a-18e04f38d7de\") " pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:08:11.672891 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.672862 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:11.672891 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.672848 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4gnx" Nov 03 15:08:11.676968 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:11.676949 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kpr4x" Nov 03 15:08:12.072509 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:12.072479 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:12.072678 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:12.072597 4557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 15:08:12.072678 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:12.072649 4557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls podName:3723d5ac-70f9-42db-a027-7f84c5d622fa nodeName:}" failed. No retries permitted until 2026-11-03 15:08:13.072636396 +0000 UTC m=+2.915757929 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x575x" (UID: "3723d5ac-70f9-42db-a027-7f84c5d622fa") : secret "insights-runtime-extractor-tls" not found Nov 03 15:08:12.076629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:12.076605 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w4gnx"] Nov 03 15:08:12.122026 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:12.121996 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2670401_d2a2_49fd_817a_18e04f38d7de.slice/crio-cdeb980b54ecd8d31d943693f21741f534f699923f42bf49021b1e9a13972ab7 WatchSource:0}: Error finding container cdeb980b54ecd8d31d943693f21741f534f699923f42bf49021b1e9a13972ab7: Status 404 returned error can't find the container with id cdeb980b54ecd8d31d943693f21741f534f699923f42bf49021b1e9a13972ab7 Nov 03 15:08:12.124141 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:12.124123 4557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:08:12.131715 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:12.131692 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kpr4x"] Nov 03 15:08:12.134700 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:12.134675 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7e4109_c1b4_4074_9c1e_cb40533b8df4.slice/crio-fc9d2544a6bf20b1fe57a8e180ba3d8826c846aa2c67ddfe1a4b0bc5eef2f549 WatchSource:0}: Error finding container fc9d2544a6bf20b1fe57a8e180ba3d8826c846aa2c67ddfe1a4b0bc5eef2f549: Status 404 returned error can't find the container with id fc9d2544a6bf20b1fe57a8e180ba3d8826c846aa2c67ddfe1a4b0bc5eef2f549 Nov 03 15:08:12.324077 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:12.323990 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w4gnx" event={"ID":"d2670401-d2a2-49fd-817a-18e04f38d7de","Type":"ContainerStarted","Data":"cdeb980b54ecd8d31d943693f21741f534f699923f42bf49021b1e9a13972ab7"} Nov 03 15:08:12.325212 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:12.325190 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kpr4x" event={"ID":"ee7e4109-c1b4-4074-9c1e-cb40533b8df4","Type":"ContainerStarted","Data":"fc9d2544a6bf20b1fe57a8e180ba3d8826c846aa2c67ddfe1a4b0bc5eef2f549"} Nov 03 15:08:12.673247 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:12.673173 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:12.873642 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:12.873611 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:13.080596 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:13.080543 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:13.080755 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:13.080714 4557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 15:08:13.080815 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:13.080792 4557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls podName:3723d5ac-70f9-42db-a027-7f84c5d622fa nodeName:}" failed. No retries permitted until 2026-11-03 15:08:15.080770804 +0000 UTC m=+4.923892347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x575x" (UID: "3723d5ac-70f9-42db-a027-7f84c5d622fa") : secret "insights-runtime-extractor-tls" not found Nov 03 15:08:13.925709 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:13.925683 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:14.337398 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:14.337351 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w4gnx" event={"ID":"d2670401-d2a2-49fd-817a-18e04f38d7de","Type":"ContainerStarted","Data":"ec0675349d861b8ad2a5db56b0e792d65b355e07b5b6dd84183b53e2c70e99e2"} Nov 03 15:08:14.337539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:14.337409 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w4gnx" event={"ID":"d2670401-d2a2-49fd-817a-18e04f38d7de","Type":"ContainerStarted","Data":"6ed46e9206655c5e8aadd58ceb941acefb4852db50519f3fb165d4631fef038a"} Nov 03 15:08:15.095423 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:15.095382 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:15.095871 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:15.095487 4557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 15:08:15.095871 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:15.095537 4557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls podName:3723d5ac-70f9-42db-a027-7f84c5d622fa nodeName:}" failed. No retries permitted until 2026-11-03 15:08:19.095523574 +0000 UTC m=+8.938645107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x575x" (UID: "3723d5ac-70f9-42db-a027-7f84c5d622fa") : secret "insights-runtime-extractor-tls" not found Nov 03 15:08:15.342476 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:15.342443 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kpr4x" event={"ID":"ee7e4109-c1b4-4074-9c1e-cb40533b8df4","Type":"ContainerStarted","Data":"100641dc027ea9141bc81caf386256f3fa4b4fd19b339b741a9380be9d5b8b42"} Nov 03 15:08:15.342647 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:15.342499 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kpr4x" event={"ID":"ee7e4109-c1b4-4074-9c1e-cb40533b8df4","Type":"ContainerStarted","Data":"17e72aa9030578cb5e6edb3bc6e4b649306d9558395d080edf0865b97e5a2fbf"} Nov 03 15:08:15.965613 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:15.965567 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:16.200083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:16.200038 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:16.346779 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:16.346750 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kpr4x" Nov 03 15:08:16.679559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:16.679481 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:16.869400 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:16.869358 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:17.088230 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:17.088200 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:17.293066 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:17.293030 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:17.476148 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:17.476081 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:17.891398 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:17.891365 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:17.891591 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:17.891565 4557 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 15:08:18.172237 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:18.172157 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:18.361852 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:18.361811 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:18.551408 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:18.551379 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:18.875493 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:18.875411 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:19.069001 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:19.068969 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:19.121642 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:19.121606 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:19.123926 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:19.123904 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3723d5ac-70f9-42db-a027-7f84c5d622fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x575x\" (UID: \"3723d5ac-70f9-42db-a027-7f84c5d622fa\") " pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:19.177213 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:19.177112 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x575x" Nov 03 15:08:19.273520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:19.273466 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:19.470043 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:19.469929 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:19.554166 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:19.554126 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x575x"] Nov 03 15:08:19.557498 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:19.557469 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3723d5ac_70f9_42db_a027_7f84c5d622fa.slice/crio-50d950a9fa110d71f015b3903c98b687400e264793b4b664a857619f8c158c97 WatchSource:0}: Error finding container 50d950a9fa110d71f015b3903c98b687400e264793b4b664a857619f8c158c97: Status 404 returned error can't find the container with id 50d950a9fa110d71f015b3903c98b687400e264793b4b664a857619f8c158c97 Nov 03 15:08:19.657009 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:19.656977 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:19.900029 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:19.899989 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:20.085514 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:20.085480 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:20.097468 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:20.097445 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:08:20.266956 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:20.266877 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:20.360480 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:20.360439 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x575x" event={"ID":"3723d5ac-70f9-42db-a027-7f84c5d622fa","Type":"ContainerStarted","Data":"11f5854c575702cd2543288dde4e05900b6b211c6234061a51d51e69d6c74fe8"} Nov 03 15:08:20.360660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:20.360500 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x575x" event={"ID":"3723d5ac-70f9-42db-a027-7f84c5d622fa","Type":"ContainerStarted","Data":"50d950a9fa110d71f015b3903c98b687400e264793b4b664a857619f8c158c97"} Nov 03 15:08:20.798102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:20.798076 4557 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-207.us-west-1.compute.internal" event="NodeReady" Nov 03 15:08:21.365115 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:21.365080 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x575x" event={"ID":"3723d5ac-70f9-42db-a027-7f84c5d622fa","Type":"ContainerStarted","Data":"3dcbafd4f9f0f7fd29b78395235733695aa0d3df56c29df0c484f0b2c44614a6"} Nov 03 15:08:21.704210 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:21.704130 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:21.899726 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:21.899693 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:23.378575 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:23.378536 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x575x" event={"ID":"3723d5ac-70f9-42db-a027-7f84c5d622fa","Type":"ContainerStarted","Data":"093a5db3cf857120e0b7df6be1b9784fe4f9902d497fdd72fd523f4f9f99185f"} Nov 03 15:08:24.519034 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.519000 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6ff7c66fd7-r5qdf"] Nov 03 15:08:24.520705 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.520689 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.561853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.561817 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm527\" (UniqueName: \"kubernetes.io/projected/b2add97e-4426-4ab3-8876-762076517b2a-kube-api-access-tm527\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.562041 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.561865 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2add97e-4426-4ab3-8876-762076517b2a-registry-certificates\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.562041 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.561924 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2add97e-4426-4ab3-8876-762076517b2a-trusted-ca\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.562041 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.561968 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2add97e-4426-4ab3-8876-762076517b2a-ca-trust-extracted\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.562041 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.561991 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2add97e-4426-4ab3-8876-762076517b2a-image-registry-private-configuration\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.562271 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.562078 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2add97e-4426-4ab3-8876-762076517b2a-registry-tls\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.562271 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.562123 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2add97e-4426-4ab3-8876-762076517b2a-bound-sa-token\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.562271 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.562221 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2add97e-4426-4ab3-8876-762076517b2a-installation-pull-secrets\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.563504 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.563445 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ff7c66fd7-r5qdf"] Nov 03 15:08:24.606007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.605969 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:24.662899 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.662870 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2add97e-4426-4ab3-8876-762076517b2a-ca-trust-extracted\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.663039 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.662906 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2add97e-4426-4ab3-8876-762076517b2a-image-registry-private-configuration\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.663039 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.662929 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2add97e-4426-4ab3-8876-762076517b2a-registry-tls\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.663039 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.662957 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2add97e-4426-4ab3-8876-762076517b2a-bound-sa-token\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.663039 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.662988 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2add97e-4426-4ab3-8876-762076517b2a-installation-pull-secrets\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.663039 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.663019 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm527\" (UniqueName: \"kubernetes.io/projected/b2add97e-4426-4ab3-8876-762076517b2a-kube-api-access-tm527\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.663305 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.663061 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2add97e-4426-4ab3-8876-762076517b2a-registry-certificates\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.663305 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.663084 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2add97e-4426-4ab3-8876-762076517b2a-trusted-ca\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.663386 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.663308 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2add97e-4426-4ab3-8876-762076517b2a-ca-trust-extracted\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.663741 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.663722 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2add97e-4426-4ab3-8876-762076517b2a-registry-certificates\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.673521 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.673500 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 15:08:24.673668 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.673536 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 15:08:24.673668 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.673614 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 15:08:24.673668 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.673643 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 15:08:24.673785 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.673687 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tqtrg\"" Nov 03 15:08:24.684075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.684054 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2add97e-4426-4ab3-8876-762076517b2a-trusted-ca\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.685312 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.685288 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2add97e-4426-4ab3-8876-762076517b2a-image-registry-private-configuration\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.685467 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.685447 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2add97e-4426-4ab3-8876-762076517b2a-installation-pull-secrets\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.685666 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.685651 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2add97e-4426-4ab3-8876-762076517b2a-registry-tls\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.724149 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.724116 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm527\" (UniqueName: \"kubernetes.io/projected/b2add97e-4426-4ab3-8876-762076517b2a-kube-api-access-tm527\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.724538 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.724516 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2add97e-4426-4ab3-8876-762076517b2a-bound-sa-token\") pod \"image-registry-6ff7c66fd7-r5qdf\" (UID: \"b2add97e-4426-4ab3-8876-762076517b2a\") " pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.795747 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.795714 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:24.829689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.829665 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:24.981505 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:24.981473 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:25.188737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:25.188655 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:25.297944 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:25.297911 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ff7c66fd7-r5qdf"] Nov 03 15:08:25.300880 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:25.300856 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2add97e_4426_4ab3_8876_762076517b2a.slice/crio-5623f2525ba7ce76433ad698e0f9ce275db9cf5b267454d3d606f18c675d97ac WatchSource:0}: Error finding container 5623f2525ba7ce76433ad698e0f9ce275db9cf5b267454d3d606f18c675d97ac: Status 404 returned error can't find the container with id 5623f2525ba7ce76433ad698e0f9ce275db9cf5b267454d3d606f18c675d97ac Nov 03 15:08:25.377368 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:25.377338 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:25.390523 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:25.390492 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" event={"ID":"b2add97e-4426-4ab3-8876-762076517b2a","Type":"ContainerStarted","Data":"c5f3c998b41639dd8eb421ac993c2c4f359cf4bb4ec3934837980ca580dc0a7c"} Nov 03 15:08:25.390673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:25.390601 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" event={"ID":"b2add97e-4426-4ab3-8876-762076517b2a","Type":"ContainerStarted","Data":"5623f2525ba7ce76433ad698e0f9ce275db9cf5b267454d3d606f18c675d97ac"} Nov 03 15:08:25.390673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:25.390629 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:25.512884 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:25.512786 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" podStartSLOduration=1.512767266 podStartE2EDuration="1.512767266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:25.51234047 +0000 UTC m=+15.355462013" watchObservedRunningTime="2026-11-03 15:08:25.512767266 +0000 UTC m=+15.355888799" Nov 03 15:08:26.352389 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:26.352360 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kpr4x" Nov 03 15:08:27.759503 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:27.759475 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mh4np" Nov 03 15:08:28.606909 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:28.606870 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:28.800036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:28.800001 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:28.814025 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:28.814002 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmvs8" Nov 03 15:08:29.038775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:29.038689 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:29.605067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:29.605037 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:29.798482 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:29.798445 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:29.994144 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:29.994069 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:33.929218 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:33.929185 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:34.120162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.120128 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:34.313522 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.313490 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:34.513736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.513703 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:34.629655 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.629564 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2jprl"] Nov 03 15:08:34.632679 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.632661 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.691807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.691771 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 15:08:34.703123 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.703093 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:34.732388 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.732355 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/241a4808-c092-4c87-b9cd-0f88efe30647-root\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.732533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.732396 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/241a4808-c092-4c87-b9cd-0f88efe30647-node-exporter-tls\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.732533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.732489 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/241a4808-c092-4c87-b9cd-0f88efe30647-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.732642 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.732537 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpskc\" (UniqueName: \"kubernetes.io/projected/241a4808-c092-4c87-b9cd-0f88efe30647-kube-api-access-kpskc\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.732642 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.732625 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/241a4808-c092-4c87-b9cd-0f88efe30647-node-exporter-textfile\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.732711 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.732657 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/241a4808-c092-4c87-b9cd-0f88efe30647-sys\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.732711 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.732677 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/241a4808-c092-4c87-b9cd-0f88efe30647-metrics-client-ca\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.732711 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.732695 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/241a4808-c092-4c87-b9cd-0f88efe30647-node-exporter-wtmp\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.732851 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.732826 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/241a4808-c092-4c87-b9cd-0f88efe30647-node-exporter-accelerators-collector-config\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.741786 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.741765 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 15:08:34.741786 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.741775 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 15:08:34.741786 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.741768 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 15:08:34.741967 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.741795 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 15:08:34.741967 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.741821 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 15:08:34.741967 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.741919 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kx5hh\"" Nov 03 15:08:34.833778 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.833746 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/241a4808-c092-4c87-b9cd-0f88efe30647-node-exporter-accelerators-collector-config\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.833951 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.833792 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/241a4808-c092-4c87-b9cd-0f88efe30647-root\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.833951 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.833816 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/241a4808-c092-4c87-b9cd-0f88efe30647-node-exporter-tls\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.833951 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.833879 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/241a4808-c092-4c87-b9cd-0f88efe30647-root\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.833951 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.833927 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/241a4808-c092-4c87-b9cd-0f88efe30647-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.834152 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.833980 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpskc\" (UniqueName: \"kubernetes.io/projected/241a4808-c092-4c87-b9cd-0f88efe30647-kube-api-access-kpskc\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.834152 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.834016 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/241a4808-c092-4c87-b9cd-0f88efe30647-node-exporter-textfile\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.834152 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.834041 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/241a4808-c092-4c87-b9cd-0f88efe30647-sys\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.834152 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.834068 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/241a4808-c092-4c87-b9cd-0f88efe30647-metrics-client-ca\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.834152 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.834092 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/241a4808-c092-4c87-b9cd-0f88efe30647-node-exporter-wtmp\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.834152 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.834135 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/241a4808-c092-4c87-b9cd-0f88efe30647-sys\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.834456 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.834241 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/241a4808-c092-4c87-b9cd-0f88efe30647-node-exporter-wtmp\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.834456 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.834352 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/241a4808-c092-4c87-b9cd-0f88efe30647-node-exporter-textfile\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.834456 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.834450 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/241a4808-c092-4c87-b9cd-0f88efe30647-node-exporter-accelerators-collector-config\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.834714 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.834669 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/241a4808-c092-4c87-b9cd-0f88efe30647-metrics-client-ca\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.836381 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.836356 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/241a4808-c092-4c87-b9cd-0f88efe30647-node-exporter-tls\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.836478 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.836359 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/241a4808-c092-4c87-b9cd-0f88efe30647-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.893141 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.893066 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:34.896073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.896050 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpskc\" (UniqueName: \"kubernetes.io/projected/241a4808-c092-4c87-b9cd-0f88efe30647-kube-api-access-kpskc\") pod \"node-exporter-2jprl\" (UID: \"241a4808-c092-4c87-b9cd-0f88efe30647\") " pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.946257 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:34.946219 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2jprl" Nov 03 15:08:34.955364 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:34.955339 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241a4808_c092_4c87_b9cd_0f88efe30647.slice/crio-fbc7b75d45d2e56e767d1971629b4bf82eb1c666ded1e6d82278dbfdae704c93 WatchSource:0}: Error finding container fbc7b75d45d2e56e767d1971629b4bf82eb1c666ded1e6d82278dbfdae704c93: Status 404 returned error can't find the container with id fbc7b75d45d2e56e767d1971629b4bf82eb1c666ded1e6d82278dbfdae704c93 Nov 03 15:08:35.123667 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:35.123621 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:35.359517 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:35.359491 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:35.421775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:35.421739 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2jprl" event={"ID":"241a4808-c092-4c87-b9cd-0f88efe30647","Type":"ContainerStarted","Data":"fbc7b75d45d2e56e767d1971629b4bf82eb1c666ded1e6d82278dbfdae704c93"} Nov 03 15:08:36.032700 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:36.032665 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:36.223347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:36.223310 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:36.428345 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:36.428313 4557 generic.go:356] "Generic (PLEG): container finished" podID="241a4808-c092-4c87-b9cd-0f88efe30647" containerID="3a1a8ea98e65f973230963ecb00f40ac2118be57dbf54168e40cc979716d8673" exitCode=0 Nov 03 15:08:36.428533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:36.428381 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2jprl" event={"ID":"241a4808-c092-4c87-b9cd-0f88efe30647","Type":"ContainerDied","Data":"3a1a8ea98e65f973230963ecb00f40ac2118be57dbf54168e40cc979716d8673"} Nov 03 15:08:36.680566 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:36.680532 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:36.886623 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:36.886568 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:37.156701 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:37.156670 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:37.434224 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:37.434138 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2jprl" event={"ID":"241a4808-c092-4c87-b9cd-0f88efe30647","Type":"ContainerStarted","Data":"93baf63c8219948e4d391c1a4dd4fe4e724219882726b16eca9675ec9d873597"} Nov 03 15:08:37.434224 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:37.434195 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2jprl" event={"ID":"241a4808-c092-4c87-b9cd-0f88efe30647","Type":"ContainerStarted","Data":"c9830a1b4fc477eedb9021dc2dba0f6298a117126ccef3a8a68a74f36a6cf1fd"} Nov 03 15:08:37.532472 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:37.532438 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:37.555155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:37.555086 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2jprl" podStartSLOduration=2.185674727 podStartE2EDuration="3.554876678s" totalImagesPullingTime="1.300785936s" totalInitContainerRuntime="68.416015ms" isStatefulPod=true podCreationTimestamp="2026-11-03 15:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:37.554820602 +0000 UTC m=+27.397942156" watchObservedRunningTime="2026-11-03 15:08:37.554876678 +0000 UTC m=+27.397998221" Nov 03 15:08:37.735730 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:37.735651 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:38.003178 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:38.003090 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:38.224838 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:38.224806 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:39.025171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.025133 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:39.236529 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.236501 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:39.451897 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.451862 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:39.641159 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.641127 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-9cdf4d6cf-kfjzh"] Nov 03 15:08:39.665303 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.665263 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:39.674156 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.674128 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.713597 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.713507 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9cdf4d6cf-kfjzh"] Nov 03 15:08:39.762162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.762130 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 15:08:39.762162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.762156 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 15:08:39.762365 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.762156 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 15:08:39.762443 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.762430 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 15:08:39.762618 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.762576 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 15:08:39.771906 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.771882 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-service-ca\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.771999 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.771913 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3608eca4-f91b-4771-b523-1c5ea739c75a-console-oauth-config\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.771999 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.771931 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-trusted-ca-bundle\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.771999 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.771957 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3608eca4-f91b-4771-b523-1c5ea739c75a-console-serving-cert\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.772129 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.772002 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-oauth-serving-cert\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.772170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.772135 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-console-config\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.772211 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.772169 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3608eca4-f91b-4771-b523-1c5ea739c75a-tmp\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.772211 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.772203 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t26l4\" (UniqueName: \"kubernetes.io/projected/3608eca4-f91b-4771-b523-1c5ea739c75a-kube-api-access-t26l4\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.777597 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.777570 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 15:08:39.777716 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.777570 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 15:08:39.777716 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.777644 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mdmnc\"" Nov 03 15:08:39.873408 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.873367 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-service-ca\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.873408 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.873404 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3608eca4-f91b-4771-b523-1c5ea739c75a-console-oauth-config\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.873663 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.873424 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-trusted-ca-bundle\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.873663 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.873444 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3608eca4-f91b-4771-b523-1c5ea739c75a-console-serving-cert\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.873663 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.873552 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-oauth-serving-cert\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.873663 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.873637 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-console-config\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.873852 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.873676 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3608eca4-f91b-4771-b523-1c5ea739c75a-tmp\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.873852 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.873714 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t26l4\" (UniqueName: \"kubernetes.io/projected/3608eca4-f91b-4771-b523-1c5ea739c75a-kube-api-access-t26l4\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.874266 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.874237 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3608eca4-f91b-4771-b523-1c5ea739c75a-tmp\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.874432 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.874390 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-service-ca\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.874496 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.874457 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-oauth-serving-cert\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.874773 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.874754 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-console-config\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.875974 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.875954 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3608eca4-f91b-4771-b523-1c5ea739c75a-console-serving-cert\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.876052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.876016 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3608eca4-f91b-4771-b523-1c5ea739c75a-console-oauth-config\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.905111 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.905078 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:39.926464 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.926426 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 15:08:39.934939 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.934906 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-trusted-ca-bundle\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.935226 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.935204 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t26l4\" (UniqueName: \"kubernetes.io/projected/3608eca4-f91b-4771-b523-1c5ea739c75a-kube-api-access-t26l4\") pod \"console-9cdf4d6cf-kfjzh\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:39.996934 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:39.996837 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:40.100335 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:40.100308 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lws9f" Nov 03 15:08:40.403385 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:40.403354 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9cdf4d6cf-kfjzh"] Nov 03 15:08:40.406215 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:40.406185 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3608eca4_f91b_4771_b523_1c5ea739c75a.slice/crio-807b26ab082b96fc5cb08962d44c7ac497cd7c3d373977218baa5704045a8b28 WatchSource:0}: Error finding container 807b26ab082b96fc5cb08962d44c7ac497cd7c3d373977218baa5704045a8b28: Status 404 returned error can't find the container with id 807b26ab082b96fc5cb08962d44c7ac497cd7c3d373977218baa5704045a8b28 Nov 03 15:08:40.446492 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:40.446454 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9cdf4d6cf-kfjzh" event={"ID":"3608eca4-f91b-4771-b523-1c5ea739c75a","Type":"ContainerStarted","Data":"807b26ab082b96fc5cb08962d44c7ac497cd7c3d373977218baa5704045a8b28"} Nov 03 15:08:41.263438 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:41.263392 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:41.452013 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:41.451970 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:44.045891 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.045860 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:44.088040 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.088012 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 15:08:44.103191 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.103166 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.156948 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.156860 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 15:08:44.157201 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.157188 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 15:08:44.206232 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.206203 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hrfg2\"" Nov 03 15:08:44.206232 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.206202 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 15:08:44.206485 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.206250 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6oq4ppcs2gt9p\"" Nov 03 15:08:44.206485 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.206260 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 15:08:44.206670 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.206654 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 15:08:44.207822 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.207793 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 15:08:44.209687 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.209666 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.209766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.209694 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.209766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.209719 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.209766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.209738 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.209862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.209799 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.209862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.209827 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.209924 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.209895 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.209957 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.209924 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbj72\" (UniqueName: \"kubernetes.io/projected/26b48fe9-4900-499f-b518-9a5f1150a0b1-kube-api-access-sbj72\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.210008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.209992 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-web-config\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.210046 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.210036 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.210076 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.210057 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.210111 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.210077 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-config\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.210111 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.210100 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.210170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.210129 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.210203 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.210180 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.210237 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.210204 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26b48fe9-4900-499f-b518-9a5f1150a0b1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.210237 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.210224 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.210294 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.210240 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26b48fe9-4900-499f-b518-9a5f1150a0b1-config-out\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.210328 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.210295 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.210357 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.210345 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.256923 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.256888 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 15:08:44.257110 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.257042 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 15:08:44.257110 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.257042 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 15:08:44.257450 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.257435 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 15:08:44.269894 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.269868 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 15:08:44.270815 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.270793 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 15:08:44.270917 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.270820 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 15:08:44.270917 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.270846 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 15:08:44.307039 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.307006 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 15:08:44.311054 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311029 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311068 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26b48fe9-4900-499f-b518-9a5f1150a0b1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311090 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311107 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26b48fe9-4900-499f-b518-9a5f1150a0b1-config-out\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311126 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311146 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311292 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311324 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311365 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311395 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311424 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311451 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311477 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311496 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbj72\" (UniqueName: \"kubernetes.io/projected/26b48fe9-4900-499f-b518-9a5f1150a0b1-kube-api-access-sbj72\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311524 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-web-config\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311550 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311594 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311973 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311631 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-config\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311973 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311655 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.311973 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.311680 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.312137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.312094 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.312305 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.312284 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.314321 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.314294 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26b48fe9-4900-499f-b518-9a5f1150a0b1-config-out\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.314436 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.314303 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.314436 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.314412 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.315078 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.314903 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.315078 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.314957 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-web-config\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.315323 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.315155 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.316815 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.315956 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.316815 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.316014 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.316815 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.316717 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.317135 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.317111 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.317261 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.317176 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.317434 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.317390 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.317636 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.317616 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26b48fe9-4900-499f-b518-9a5f1150a0b1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.317636 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.317631 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.318978 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.318956 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.319119 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.319103 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-config\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.334007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.333979 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 15:08:44.342728 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.342706 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.379327 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.379294 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbj72\" (UniqueName: \"kubernetes.io/projected/26b48fe9-4900-499f-b518-9a5f1150a0b1-kube-api-access-sbj72\") pod \"prometheus-k8s-0\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.461609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.461515 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9cdf4d6cf-kfjzh" event={"ID":"3608eca4-f91b-4771-b523-1c5ea739c75a","Type":"ContainerStarted","Data":"399400ca7be7466fca66f52901b2bd23b56adc4c9ef46f8b2357b1183528bb4b"} Nov 03 15:08:44.499044 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.499009 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:08:44.961861 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.961800 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-9cdf4d6cf-kfjzh" podStartSLOduration=2.462324364 podStartE2EDuration="5.961780204s" totalImagesPullingTime="3.49945584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:44.582771711 +0000 UTC m=+34.425893252" watchObservedRunningTime="2026-11-03 15:08:44.961780204 +0000 UTC m=+34.804901742" Nov 03 15:08:44.962246 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:44.962227 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 15:08:44.965372 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:08:44.965343 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b48fe9_4900_499f_b518_9a5f1150a0b1.slice/crio-5083c425da0faa12b3f7e4372898a04f5b1de50f64c7da1208c79a7f891f0b22 WatchSource:0}: Error finding container 5083c425da0faa12b3f7e4372898a04f5b1de50f64c7da1208c79a7f891f0b22: Status 404 returned error can't find the container with id 5083c425da0faa12b3f7e4372898a04f5b1de50f64c7da1208c79a7f891f0b22 Nov 03 15:08:45.466302 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:45.466270 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b48fe9-4900-499f-b518-9a5f1150a0b1","Type":"ContainerStarted","Data":"5083c425da0faa12b3f7e4372898a04f5b1de50f64c7da1208c79a7f891f0b22"} Nov 03 15:08:46.399013 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:46.398981 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6ff7c66fd7-r5qdf" Nov 03 15:08:46.528035 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:08:46.528001 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:08:46.758338 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:46.758262 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:46.959781 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:46.959753 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:47.066593 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:47.066559 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9cdf4d6cf-kfjzh"] Nov 03 15:08:47.190866 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:47.190831 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:47.474467 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:47.474388 4557 generic.go:356] "Generic (PLEG): container finished" podID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerID="87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a" exitCode=0 Nov 03 15:08:47.474467 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:47.474433 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b48fe9-4900-499f-b518-9a5f1150a0b1","Type":"ContainerDied","Data":"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a"} Nov 03 15:08:48.442088 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:48.442048 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:48.632877 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:48.632841 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:49.105471 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:49.105436 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:49.298645 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:49.298615 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:49.997204 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:49.997170 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:08:51.371817 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:51.371729 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:51.491056 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:51.491022 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b48fe9-4900-499f-b518-9a5f1150a0b1","Type":"ContainerStarted","Data":"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713"} Nov 03 15:08:51.491202 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:51.491087 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b48fe9-4900-499f-b518-9a5f1150a0b1","Type":"ContainerStarted","Data":"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c"} Nov 03 15:08:51.665198 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:51.665126 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:54.504715 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:54.504674 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b48fe9-4900-499f-b518-9a5f1150a0b1","Type":"ContainerStarted","Data":"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025"} Nov 03 15:08:54.505193 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:54.504753 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b48fe9-4900-499f-b518-9a5f1150a0b1","Type":"ContainerStarted","Data":"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9"} Nov 03 15:08:54.505193 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:54.504793 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b48fe9-4900-499f-b518-9a5f1150a0b1","Type":"ContainerStarted","Data":"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37"} Nov 03 15:08:54.505193 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:54.504845 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b48fe9-4900-499f-b518-9a5f1150a0b1","Type":"ContainerStarted","Data":"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984"} Nov 03 15:08:54.690794 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:54.690739 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.926889316 podStartE2EDuration="11.690726096s" totalImagesPullingTime="7.737876443s" totalInitContainerRuntime="25.960337ms" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:54.689466973 +0000 UTC m=+44.532588550" watchObservedRunningTime="2026-11-03 15:08:54.690726096 +0000 UTC m=+44.533847639" Nov 03 15:08:54.999603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:54.999552 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:55.276045 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:55.275963 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:58.060890 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:58.060852 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:08:59.500027 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:08:59.499993 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:09:00.507441 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:00.507410 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:09:02.251520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:02.251489 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:09:02.909209 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:02.909173 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:09:03.104023 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:03.103992 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:09:09.806503 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:09.806468 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:09:09.995913 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:09.995883 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:09:12.085809 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.085772 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-9cdf4d6cf-kfjzh" podUID="3608eca4-f91b-4771-b523-1c5ea739c75a" containerName="console" containerID="cri-o://399400ca7be7466fca66f52901b2bd23b56adc4c9ef46f8b2357b1183528bb4b" gracePeriod=15 Nov 03 15:09:12.341853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.341786 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:09:12.430810 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.430771 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-console-config\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-console-config\") pod \"3608eca4-f91b-4771-b523-1c5ea739c75a\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " Nov 03 15:09:12.431004 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.430874 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3608eca4-f91b-4771-b523-1c5ea739c75a-kube-api-access-t26l4\" (UniqueName: \"kubernetes.io/projected/3608eca4-f91b-4771-b523-1c5ea739c75a-kube-api-access-t26l4\") pod \"3608eca4-f91b-4771-b523-1c5ea739c75a\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " Nov 03 15:09:12.431004 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.430910 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3608eca4-f91b-4771-b523-1c5ea739c75a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3608eca4-f91b-4771-b523-1c5ea739c75a-console-oauth-config\") pod \"3608eca4-f91b-4771-b523-1c5ea739c75a\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " Nov 03 15:09:12.431004 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.430952 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3608eca4-f91b-4771-b523-1c5ea739c75a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3608eca4-f91b-4771-b523-1c5ea739c75a-console-serving-cert\") pod \"3608eca4-f91b-4771-b523-1c5ea739c75a\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " Nov 03 15:09:12.431177 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.431005 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3608eca4-f91b-4771-b523-1c5ea739c75a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3608eca4-f91b-4771-b523-1c5ea739c75a-tmp\") pod \"3608eca4-f91b-4771-b523-1c5ea739c75a\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " Nov 03 15:09:12.431177 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.431041 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-service-ca\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-service-ca\") pod \"3608eca4-f91b-4771-b523-1c5ea739c75a\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " Nov 03 15:09:12.431177 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.431068 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-trusted-ca-bundle\") pod \"3608eca4-f91b-4771-b523-1c5ea739c75a\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " Nov 03 15:09:12.431177 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.431158 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-console-config" pod "3608eca4-f91b-4771-b523-1c5ea739c75a" (UID: "3608eca4-f91b-4771-b523-1c5ea739c75a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:09:12.431388 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.431275 4557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-console-config\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:12.431388 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.431297 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3608eca4-f91b-4771-b523-1c5ea739c75a-tmp" pod "3608eca4-f91b-4771-b523-1c5ea739c75a" (UID: "3608eca4-f91b-4771-b523-1c5ea739c75a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:09:12.431494 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.431445 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-service-ca" pod "3608eca4-f91b-4771-b523-1c5ea739c75a" (UID: "3608eca4-f91b-4771-b523-1c5ea739c75a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:09:12.431615 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.431563 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-trusted-ca-bundle" pod "3608eca4-f91b-4771-b523-1c5ea739c75a" (UID: "3608eca4-f91b-4771-b523-1c5ea739c75a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:09:12.433198 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.433169 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3608eca4-f91b-4771-b523-1c5ea739c75a-console-serving-cert" pod "3608eca4-f91b-4771-b523-1c5ea739c75a" (UID: "3608eca4-f91b-4771-b523-1c5ea739c75a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:09:12.433294 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.433202 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3608eca4-f91b-4771-b523-1c5ea739c75a-kube-api-access-t26l4" pod "3608eca4-f91b-4771-b523-1c5ea739c75a" (UID: "3608eca4-f91b-4771-b523-1c5ea739c75a"). InnerVolumeSpecName "kube-api-access-t26l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:09:12.433294 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.433256 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3608eca4-f91b-4771-b523-1c5ea739c75a-console-oauth-config" pod "3608eca4-f91b-4771-b523-1c5ea739c75a" (UID: "3608eca4-f91b-4771-b523-1c5ea739c75a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:09:12.531793 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.531759 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-oauth-serving-cert\") pod \"3608eca4-f91b-4771-b523-1c5ea739c75a\" (UID: \"3608eca4-f91b-4771-b523-1c5ea739c75a\") " Nov 03 15:09:12.531964 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.531941 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t26l4\" (UniqueName: \"kubernetes.io/projected/3608eca4-f91b-4771-b523-1c5ea739c75a-kube-api-access-t26l4\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:12.531964 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.531954 4557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3608eca4-f91b-4771-b523-1c5ea739c75a-console-oauth-config\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:12.531964 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.531963 4557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3608eca4-f91b-4771-b523-1c5ea739c75a-console-serving-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:12.532073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.531973 4557 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3608eca4-f91b-4771-b523-1c5ea739c75a-tmp\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:12.532073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.531982 4557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-service-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:12.532073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.531990 4557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-trusted-ca-bundle\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:12.532174 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.532083 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-oauth-serving-cert" pod "3608eca4-f91b-4771-b523-1c5ea739c75a" (UID: "3608eca4-f91b-4771-b523-1c5ea739c75a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:09:12.567786 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.567751 4557 generic.go:356] "Generic (PLEG): container finished" podID="3608eca4-f91b-4771-b523-1c5ea739c75a" containerID="399400ca7be7466fca66f52901b2bd23b56adc4c9ef46f8b2357b1183528bb4b" exitCode=0 Nov 03 15:09:12.567786 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.567790 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9cdf4d6cf-kfjzh" event={"ID":"3608eca4-f91b-4771-b523-1c5ea739c75a","Type":"ContainerDied","Data":"399400ca7be7466fca66f52901b2bd23b56adc4c9ef46f8b2357b1183528bb4b"} Nov 03 15:09:12.567977 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.567819 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9cdf4d6cf-kfjzh" Nov 03 15:09:12.567977 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.567856 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9cdf4d6cf-kfjzh" event={"ID":"3608eca4-f91b-4771-b523-1c5ea739c75a","Type":"ContainerDied","Data":"807b26ab082b96fc5cb08962d44c7ac497cd7c3d373977218baa5704045a8b28"} Nov 03 15:09:12.567977 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.567885 4557 scope.go:122] "RemoveContainer" containerID="399400ca7be7466fca66f52901b2bd23b56adc4c9ef46f8b2357b1183528bb4b" Nov 03 15:09:12.576960 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.576935 4557 scope.go:122] "RemoveContainer" containerID="399400ca7be7466fca66f52901b2bd23b56adc4c9ef46f8b2357b1183528bb4b" Nov 03 15:09:12.577222 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:09:12.577203 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"399400ca7be7466fca66f52901b2bd23b56adc4c9ef46f8b2357b1183528bb4b\": container with ID starting with 399400ca7be7466fca66f52901b2bd23b56adc4c9ef46f8b2357b1183528bb4b not found: ID does not exist" containerID="399400ca7be7466fca66f52901b2bd23b56adc4c9ef46f8b2357b1183528bb4b" Nov 03 15:09:12.577274 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.577233 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399400ca7be7466fca66f52901b2bd23b56adc4c9ef46f8b2357b1183528bb4b"} err="failed to get container status \"399400ca7be7466fca66f52901b2bd23b56adc4c9ef46f8b2357b1183528bb4b\": rpc error: code = NotFound desc = could not find container \"399400ca7be7466fca66f52901b2bd23b56adc4c9ef46f8b2357b1183528bb4b\": container with ID starting with 399400ca7be7466fca66f52901b2bd23b56adc4c9ef46f8b2357b1183528bb4b not found: ID does not exist" Nov 03 15:09:12.632426 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.632350 4557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3608eca4-f91b-4771-b523-1c5ea739c75a-oauth-serving-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:12.737947 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.737915 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9cdf4d6cf-kfjzh"] Nov 03 15:09:12.742738 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:12.742715 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9cdf4d6cf-kfjzh"] Nov 03 15:09:14.282728 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:14.282697 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3608eca4-f91b-4771-b523-1c5ea739c75a" path="/var/lib/kubelet/pods/3608eca4-f91b-4771-b523-1c5ea739c75a/volumes" Nov 03 15:09:16.528874 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:09:16.528838 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:09:16.668410 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:16.668380 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:09:16.856558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:16.856527 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:09:30.628314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:30.628240 4557 generic.go:356] "Generic (PLEG): container finished" podID="450408b0-4d59-4e6d-a643-a2d1e15e459e" containerID="22d8d72aa349c9a661dce61170ccedcf5bc970fbb285922774eb1ac43e8c4856" exitCode=0 Nov 03 15:09:30.628712 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:30.628315 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" event={"ID":"450408b0-4d59-4e6d-a643-a2d1e15e459e","Type":"ContainerDied","Data":"22d8d72aa349c9a661dce61170ccedcf5bc970fbb285922774eb1ac43e8c4856"} Nov 03 15:09:31.766159 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:31.766134 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:09:31.870490 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:31.870452 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450408b0-4d59-4e6d-a643-a2d1e15e459e-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/450408b0-4d59-4e6d-a643-a2d1e15e459e-archives-path\") pod \"450408b0-4d59-4e6d-a643-a2d1e15e459e\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " Nov 03 15:09:31.870704 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:31.870503 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-service-ca-bundle\") pod \"450408b0-4d59-4e6d-a643-a2d1e15e459e\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " Nov 03 15:09:31.870704 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:31.870553 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-trusted-ca-bundle\") pod \"450408b0-4d59-4e6d-a643-a2d1e15e459e\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " Nov 03 15:09:31.870704 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:31.870637 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450408b0-4d59-4e6d-a643-a2d1e15e459e-kube-api-access-fhhfg\" (UniqueName: \"kubernetes.io/projected/450408b0-4d59-4e6d-a643-a2d1e15e459e-kube-api-access-fhhfg\") pod \"450408b0-4d59-4e6d-a643-a2d1e15e459e\" (UID: \"450408b0-4d59-4e6d-a643-a2d1e15e459e\") " Nov 03 15:09:31.870848 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:31.870813 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450408b0-4d59-4e6d-a643-a2d1e15e459e-archives-path" pod "450408b0-4d59-4e6d-a643-a2d1e15e459e" (UID: "450408b0-4d59-4e6d-a643-a2d1e15e459e"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:09:31.870951 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:31.870930 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-service-ca-bundle" pod "450408b0-4d59-4e6d-a643-a2d1e15e459e" (UID: "450408b0-4d59-4e6d-a643-a2d1e15e459e"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:09:31.871071 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:31.871049 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-trusted-ca-bundle" pod "450408b0-4d59-4e6d-a643-a2d1e15e459e" (UID: "450408b0-4d59-4e6d-a643-a2d1e15e459e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:09:31.872911 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:31.872887 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450408b0-4d59-4e6d-a643-a2d1e15e459e-kube-api-access-fhhfg" pod "450408b0-4d59-4e6d-a643-a2d1e15e459e" (UID: "450408b0-4d59-4e6d-a643-a2d1e15e459e"). InnerVolumeSpecName "kube-api-access-fhhfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:09:31.971759 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:31.971672 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhhfg\" (UniqueName: \"kubernetes.io/projected/450408b0-4d59-4e6d-a643-a2d1e15e459e-kube-api-access-fhhfg\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:31.971759 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:31.971703 4557 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/450408b0-4d59-4e6d-a643-a2d1e15e459e-archives-path\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:31.971759 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:31.971716 4557 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-service-ca-bundle\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:31.971759 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:31.971725 4557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450408b0-4d59-4e6d-a643-a2d1e15e459e-trusted-ca-bundle\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:09:32.636667 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:32.636633 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" event={"ID":"450408b0-4d59-4e6d-a643-a2d1e15e459e","Type":"ContainerDied","Data":"e26ac501b180bf89482d47c694bb618b1414817437a6b6bf005586eaaeb9f23a"} Nov 03 15:09:32.636840 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:32.636681 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e26ac501b180bf89482d47c694bb618b1414817437a6b6bf005586eaaeb9f23a" Nov 03 15:09:32.636840 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:32.636642 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hwzmv-gkbt4" Nov 03 15:09:33.727497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:33.727459 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:09:33.916939 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:33.916907 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:09:43.635032 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:43.634995 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:09:43.823724 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:43.823676 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:09:46.528986 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:09:46.528940 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:09:49.211227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:49.211168 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:09:49.229233 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:49.229206 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:09:49.515123 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:49.515049 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:09:53.715164 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:53.715130 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:09:53.932617 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:53.932572 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:09:55.369152 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:55.369114 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:09:56.467527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:56.467491 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:09:56.658616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:56.658570 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:09:56.879050 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:09:56.879010 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:10:06.435625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:06.435575 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:10:06.619836 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:06.619791 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:10:07.207879 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.207844 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 15:10:07.208271 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.208161 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="prometheus" containerID="cri-o://0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c" gracePeriod=600 Nov 03 15:10:07.208271 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.208228 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="kube-rbac-proxy" containerID="cri-o://eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9" gracePeriod=600 Nov 03 15:10:07.208271 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.208215 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="kube-rbac-proxy-thanos" containerID="cri-o://3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025" gracePeriod=600 Nov 03 15:10:07.208534 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.208220 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="config-reloader" containerID="cri-o://f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713" gracePeriod=600 Nov 03 15:10:07.208534 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.208231 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="kube-rbac-proxy-web" containerID="cri-o://12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37" gracePeriod=600 Nov 03 15:10:07.208534 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.208229 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="thanos-sidecar" containerID="cri-o://bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984" gracePeriod=600 Nov 03 15:10:07.450874 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.450838 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:07.547796 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.547764 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-0\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.547966 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.547811 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-thanos-prometheus-http-client-file\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.547966 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.547838 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-metrics-client-certs\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.547966 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.547873 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-config\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-config\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.547966 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.547911 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.547966 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.547941 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-kube-rbac-proxy\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.548243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.548002 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-kubelet-serving-ca-bundle\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.548243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.548030 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-2\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.548243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.548058 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-grpc-tls\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.548243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.548086 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26b48fe9-4900-499f-b518-9a5f1150a0b1-config-out\" (UniqueName: \"kubernetes.io/empty-dir/26b48fe9-4900-499f-b518-9a5f1150a0b1-config-out\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.548243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.548117 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-db\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.548243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.548143 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b48fe9-4900-499f-b518-9a5f1150a0b1-tls-assets\" (UniqueName: \"kubernetes.io/projected/26b48fe9-4900-499f-b518-9a5f1150a0b1-tls-assets\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.548243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.548173 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-serving-certs-ca-bundle\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.548243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.548203 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-tls\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.548243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.548238 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.548777 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.548280 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-1\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.548777 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.548327 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-metrics-client-ca\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.548777 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.548354 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-trusted-ca-bundle\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.548777 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.548383 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b48fe9-4900-499f-b518-9a5f1150a0b1-kube-api-access-sbj72\" (UniqueName: \"kubernetes.io/projected/26b48fe9-4900-499f-b518-9a5f1150a0b1-kube-api-access-sbj72\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.548777 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.548408 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-web-config\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-web-config\") pod \"26b48fe9-4900-499f-b518-9a5f1150a0b1\" (UID: \"26b48fe9-4900-499f-b518-9a5f1150a0b1\") " Nov 03 15:10:07.549601 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.549311 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-db" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:10:07.549726 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.549700 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-0" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:10:07.549796 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.549763 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-kubelet-serving-ca-bundle" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:10:07.550713 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.550337 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-1" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:10:07.550713 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.550689 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-metrics-client-ca" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:10:07.550905 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.550716 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-config" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:10:07.550905 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.550782 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-metrics-client-certs" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:10:07.551206 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.551100 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-trusted-ca-bundle" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:10:07.551206 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.551106 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-serving-certs-ca-bundle" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:10:07.551206 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.551167 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-2" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:10:07.551206 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.551180 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-thanos-prometheus-http-client-file" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:10:07.551610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.551543 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-kube-rbac-proxy-web" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:10:07.552060 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.552032 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-kube-rbac-proxy" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:10:07.552861 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.552833 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-thanos-sidecar-tls" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:10:07.552992 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.552973 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b48fe9-4900-499f-b518-9a5f1150a0b1-kube-api-access-sbj72" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "kube-api-access-sbj72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:10:07.553422 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.553401 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b48fe9-4900-499f-b518-9a5f1150a0b1-tls-assets" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:10:07.553515 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.553496 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-grpc-tls" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:10:07.553620 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.553595 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26b48fe9-4900-499f-b518-9a5f1150a0b1-config-out" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:10:07.553678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.553618 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-tls" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:10:07.561290 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.561270 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-web-config" pod "26b48fe9-4900-499f-b518-9a5f1150a0b1" (UID: "26b48fe9-4900-499f-b518-9a5f1150a0b1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:10:07.649349 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649312 4557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649349 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649343 4557 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-metrics-client-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649349 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649354 4557 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649365 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbj72\" (UniqueName: \"kubernetes.io/projected/26b48fe9-4900-499f-b518-9a5f1150a0b1-kube-api-access-sbj72\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649396 4557 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-web-config\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649406 4557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649421 4557 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649431 4557 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-metrics-client-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649439 4557 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-config\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649448 4557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649456 4557 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-kube-rbac-proxy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649466 4557 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649475 4557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649489 4557 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-grpc-tls\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649501 4557 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26b48fe9-4900-499f-b518-9a5f1150a0b1-config-out\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649510 4557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/26b48fe9-4900-499f-b518-9a5f1150a0b1-prometheus-k8s-db\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649518 4557 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26b48fe9-4900-499f-b518-9a5f1150a0b1-tls-assets\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649526 4557 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b48fe9-4900-499f-b518-9a5f1150a0b1-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649534 4557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.649558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.649543 4557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/26b48fe9-4900-499f-b518-9a5f1150a0b1-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:10:07.755120 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.755088 4557 generic.go:356] "Generic (PLEG): container finished" podID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerID="3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025" exitCode=0 Nov 03 15:10:07.755120 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.755112 4557 generic.go:356] "Generic (PLEG): container finished" podID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerID="eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9" exitCode=0 Nov 03 15:10:07.755120 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.755118 4557 generic.go:356] "Generic (PLEG): container finished" podID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerID="12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37" exitCode=0 Nov 03 15:10:07.755120 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.755124 4557 generic.go:356] "Generic (PLEG): container finished" podID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerID="bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984" exitCode=0 Nov 03 15:10:07.755120 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.755129 4557 generic.go:356] "Generic (PLEG): container finished" podID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerID="f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713" exitCode=0 Nov 03 15:10:07.755405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.755136 4557 generic.go:356] "Generic (PLEG): container finished" podID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerID="0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c" exitCode=0 Nov 03 15:10:07.755405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.755170 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:07.755405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.755177 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b48fe9-4900-499f-b518-9a5f1150a0b1","Type":"ContainerDied","Data":"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025"} Nov 03 15:10:07.755405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.755250 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b48fe9-4900-499f-b518-9a5f1150a0b1","Type":"ContainerDied","Data":"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9"} Nov 03 15:10:07.755405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.755278 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b48fe9-4900-499f-b518-9a5f1150a0b1","Type":"ContainerDied","Data":"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37"} Nov 03 15:10:07.755405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.755303 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b48fe9-4900-499f-b518-9a5f1150a0b1","Type":"ContainerDied","Data":"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984"} Nov 03 15:10:07.755405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.755322 4557 scope.go:122] "RemoveContainer" containerID="3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025" Nov 03 15:10:07.755405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.755327 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b48fe9-4900-499f-b518-9a5f1150a0b1","Type":"ContainerDied","Data":"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713"} Nov 03 15:10:07.755405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.755353 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b48fe9-4900-499f-b518-9a5f1150a0b1","Type":"ContainerDied","Data":"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c"} Nov 03 15:10:07.755405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.755377 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26b48fe9-4900-499f-b518-9a5f1150a0b1","Type":"ContainerDied","Data":"5083c425da0faa12b3f7e4372898a04f5b1de50f64c7da1208c79a7f891f0b22"} Nov 03 15:10:07.764066 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.764045 4557 scope.go:122] "RemoveContainer" containerID="eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9" Nov 03 15:10:07.772389 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.772360 4557 scope.go:122] "RemoveContainer" containerID="12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37" Nov 03 15:10:07.779005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.778985 4557 scope.go:122] "RemoveContainer" containerID="bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984" Nov 03 15:10:07.785239 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.785222 4557 scope.go:122] "RemoveContainer" containerID="f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713" Nov 03 15:10:07.791364 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.791346 4557 scope.go:122] "RemoveContainer" containerID="0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c" Nov 03 15:10:07.798038 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.798023 4557 scope.go:122] "RemoveContainer" containerID="87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a" Nov 03 15:10:07.804196 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.804178 4557 scope.go:122] "RemoveContainer" containerID="3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025" Nov 03 15:10:07.804459 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:10:07.804439 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025\": container with ID starting with 3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025 not found: ID does not exist" containerID="3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025" Nov 03 15:10:07.804527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.804475 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025"} err="failed to get container status \"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025\": rpc error: code = NotFound desc = could not find container \"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025\": container with ID starting with 3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025 not found: ID does not exist" Nov 03 15:10:07.804527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.804503 4557 scope.go:122] "RemoveContainer" containerID="eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9" Nov 03 15:10:07.804744 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:10:07.804728 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9\": container with ID starting with eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9 not found: ID does not exist" containerID="eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9" Nov 03 15:10:07.804780 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.804754 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9"} err="failed to get container status \"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9\": rpc error: code = NotFound desc = could not find container \"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9\": container with ID starting with eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9 not found: ID does not exist" Nov 03 15:10:07.804780 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.804770 4557 scope.go:122] "RemoveContainer" containerID="12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37" Nov 03 15:10:07.805005 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:10:07.804988 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37\": container with ID starting with 12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37 not found: ID does not exist" containerID="12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37" Nov 03 15:10:07.805067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.805015 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37"} err="failed to get container status \"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37\": rpc error: code = NotFound desc = could not find container \"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37\": container with ID starting with 12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37 not found: ID does not exist" Nov 03 15:10:07.805067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.805037 4557 scope.go:122] "RemoveContainer" containerID="bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984" Nov 03 15:10:07.805315 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:10:07.805297 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984\": container with ID starting with bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984 not found: ID does not exist" containerID="bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984" Nov 03 15:10:07.805352 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.805321 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984"} err="failed to get container status \"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984\": rpc error: code = NotFound desc = could not find container \"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984\": container with ID starting with bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984 not found: ID does not exist" Nov 03 15:10:07.805352 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.805335 4557 scope.go:122] "RemoveContainer" containerID="f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713" Nov 03 15:10:07.805544 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:10:07.805527 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713\": container with ID starting with f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713 not found: ID does not exist" containerID="f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713" Nov 03 15:10:07.805645 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.805627 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713"} err="failed to get container status \"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713\": rpc error: code = NotFound desc = could not find container \"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713\": container with ID starting with f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713 not found: ID does not exist" Nov 03 15:10:07.805712 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.805648 4557 scope.go:122] "RemoveContainer" containerID="0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c" Nov 03 15:10:07.805882 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:10:07.805860 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c\": container with ID starting with 0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c not found: ID does not exist" containerID="0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c" Nov 03 15:10:07.805926 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.805889 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c"} err="failed to get container status \"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c\": rpc error: code = NotFound desc = could not find container \"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c\": container with ID starting with 0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c not found: ID does not exist" Nov 03 15:10:07.805926 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.805903 4557 scope.go:122] "RemoveContainer" containerID="87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a" Nov 03 15:10:07.806076 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:10:07.806062 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a\": container with ID starting with 87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a not found: ID does not exist" containerID="87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a" Nov 03 15:10:07.806115 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.806079 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a"} err="failed to get container status \"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a\": rpc error: code = NotFound desc = could not find container \"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a\": container with ID starting with 87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a not found: ID does not exist" Nov 03 15:10:07.806115 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.806090 4557 scope.go:122] "RemoveContainer" containerID="3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025" Nov 03 15:10:07.806242 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.806228 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025"} err="failed to get container status \"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025\": rpc error: code = NotFound desc = could not find container \"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025\": container with ID starting with 3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025 not found: ID does not exist" Nov 03 15:10:07.806283 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.806241 4557 scope.go:122] "RemoveContainer" containerID="eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9" Nov 03 15:10:07.806448 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.806429 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9"} err="failed to get container status \"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9\": rpc error: code = NotFound desc = could not find container \"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9\": container with ID starting with eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9 not found: ID does not exist" Nov 03 15:10:07.806516 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.806449 4557 scope.go:122] "RemoveContainer" containerID="12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37" Nov 03 15:10:07.806684 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.806667 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37"} err="failed to get container status \"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37\": rpc error: code = NotFound desc = could not find container \"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37\": container with ID starting with 12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37 not found: ID does not exist" Nov 03 15:10:07.806684 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.806683 4557 scope.go:122] "RemoveContainer" containerID="bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984" Nov 03 15:10:07.806884 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.806867 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984"} err="failed to get container status \"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984\": rpc error: code = NotFound desc = could not find container \"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984\": container with ID starting with bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984 not found: ID does not exist" Nov 03 15:10:07.806926 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.806884 4557 scope.go:122] "RemoveContainer" containerID="f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713" Nov 03 15:10:07.807072 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.807056 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713"} err="failed to get container status \"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713\": rpc error: code = NotFound desc = could not find container \"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713\": container with ID starting with f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713 not found: ID does not exist" Nov 03 15:10:07.807112 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.807071 4557 scope.go:122] "RemoveContainer" containerID="0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c" Nov 03 15:10:07.807258 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.807242 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c"} err="failed to get container status \"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c\": rpc error: code = NotFound desc = could not find container \"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c\": container with ID starting with 0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c not found: ID does not exist" Nov 03 15:10:07.807295 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.807257 4557 scope.go:122] "RemoveContainer" containerID="87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a" Nov 03 15:10:07.807437 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.807422 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a"} err="failed to get container status \"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a\": rpc error: code = NotFound desc = could not find container \"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a\": container with ID starting with 87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a not found: ID does not exist" Nov 03 15:10:07.807481 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.807437 4557 scope.go:122] "RemoveContainer" containerID="3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025" Nov 03 15:10:07.807618 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.807601 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025"} err="failed to get container status \"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025\": rpc error: code = NotFound desc = could not find container \"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025\": container with ID starting with 3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025 not found: ID does not exist" Nov 03 15:10:07.807618 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.807616 4557 scope.go:122] "RemoveContainer" containerID="eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9" Nov 03 15:10:07.807798 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.807782 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9"} err="failed to get container status \"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9\": rpc error: code = NotFound desc = could not find container \"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9\": container with ID starting with eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9 not found: ID does not exist" Nov 03 15:10:07.807855 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.807798 4557 scope.go:122] "RemoveContainer" containerID="12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37" Nov 03 15:10:07.807978 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.807964 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37"} err="failed to get container status \"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37\": rpc error: code = NotFound desc = could not find container \"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37\": container with ID starting with 12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37 not found: ID does not exist" Nov 03 15:10:07.808019 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.807981 4557 scope.go:122] "RemoveContainer" containerID="bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984" Nov 03 15:10:07.808164 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.808142 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984"} err="failed to get container status \"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984\": rpc error: code = NotFound desc = could not find container \"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984\": container with ID starting with bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984 not found: ID does not exist" Nov 03 15:10:07.808231 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.808164 4557 scope.go:122] "RemoveContainer" containerID="f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713" Nov 03 15:10:07.808376 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.808358 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713"} err="failed to get container status \"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713\": rpc error: code = NotFound desc = could not find container \"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713\": container with ID starting with f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713 not found: ID does not exist" Nov 03 15:10:07.808376 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.808375 4557 scope.go:122] "RemoveContainer" containerID="0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c" Nov 03 15:10:07.808573 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.808552 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c"} err="failed to get container status \"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c\": rpc error: code = NotFound desc = could not find container \"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c\": container with ID starting with 0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c not found: ID does not exist" Nov 03 15:10:07.808573 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.808571 4557 scope.go:122] "RemoveContainer" containerID="87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a" Nov 03 15:10:07.808824 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.808807 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a"} err="failed to get container status \"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a\": rpc error: code = NotFound desc = could not find container \"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a\": container with ID starting with 87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a not found: ID does not exist" Nov 03 15:10:07.808888 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.808830 4557 scope.go:122] "RemoveContainer" containerID="3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025" Nov 03 15:10:07.809041 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.809019 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025"} err="failed to get container status \"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025\": rpc error: code = NotFound desc = could not find container \"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025\": container with ID starting with 3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025 not found: ID does not exist" Nov 03 15:10:07.809101 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.809041 4557 scope.go:122] "RemoveContainer" containerID="eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9" Nov 03 15:10:07.809243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.809223 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9"} err="failed to get container status \"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9\": rpc error: code = NotFound desc = could not find container \"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9\": container with ID starting with eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9 not found: ID does not exist" Nov 03 15:10:07.809301 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.809243 4557 scope.go:122] "RemoveContainer" containerID="12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37" Nov 03 15:10:07.809433 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.809418 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37"} err="failed to get container status \"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37\": rpc error: code = NotFound desc = could not find container \"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37\": container with ID starting with 12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37 not found: ID does not exist" Nov 03 15:10:07.809489 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.809432 4557 scope.go:122] "RemoveContainer" containerID="bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984" Nov 03 15:10:07.809621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.809605 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984"} err="failed to get container status \"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984\": rpc error: code = NotFound desc = could not find container \"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984\": container with ID starting with bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984 not found: ID does not exist" Nov 03 15:10:07.809621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.809619 4557 scope.go:122] "RemoveContainer" containerID="f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713" Nov 03 15:10:07.809818 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.809800 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713"} err="failed to get container status \"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713\": rpc error: code = NotFound desc = could not find container \"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713\": container with ID starting with f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713 not found: ID does not exist" Nov 03 15:10:07.809870 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.809818 4557 scope.go:122] "RemoveContainer" containerID="0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c" Nov 03 15:10:07.810046 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.810027 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c"} err="failed to get container status \"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c\": rpc error: code = NotFound desc = could not find container \"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c\": container with ID starting with 0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c not found: ID does not exist" Nov 03 15:10:07.810046 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.810043 4557 scope.go:122] "RemoveContainer" containerID="87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a" Nov 03 15:10:07.810259 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.810237 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a"} err="failed to get container status \"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a\": rpc error: code = NotFound desc = could not find container \"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a\": container with ID starting with 87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a not found: ID does not exist" Nov 03 15:10:07.810332 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.810260 4557 scope.go:122] "RemoveContainer" containerID="3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025" Nov 03 15:10:07.810462 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.810444 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025"} err="failed to get container status \"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025\": rpc error: code = NotFound desc = could not find container \"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025\": container with ID starting with 3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025 not found: ID does not exist" Nov 03 15:10:07.810509 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.810462 4557 scope.go:122] "RemoveContainer" containerID="eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9" Nov 03 15:10:07.810723 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.810707 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9"} err="failed to get container status \"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9\": rpc error: code = NotFound desc = could not find container \"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9\": container with ID starting with eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9 not found: ID does not exist" Nov 03 15:10:07.810759 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.810723 4557 scope.go:122] "RemoveContainer" containerID="12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37" Nov 03 15:10:07.810925 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.810909 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37"} err="failed to get container status \"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37\": rpc error: code = NotFound desc = could not find container \"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37\": container with ID starting with 12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37 not found: ID does not exist" Nov 03 15:10:07.810969 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.810925 4557 scope.go:122] "RemoveContainer" containerID="bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984" Nov 03 15:10:07.811109 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.811090 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984"} err="failed to get container status \"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984\": rpc error: code = NotFound desc = could not find container \"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984\": container with ID starting with bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984 not found: ID does not exist" Nov 03 15:10:07.811149 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.811109 4557 scope.go:122] "RemoveContainer" containerID="f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713" Nov 03 15:10:07.811271 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.811254 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713"} err="failed to get container status \"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713\": rpc error: code = NotFound desc = could not find container \"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713\": container with ID starting with f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713 not found: ID does not exist" Nov 03 15:10:07.811316 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.811272 4557 scope.go:122] "RemoveContainer" containerID="0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c" Nov 03 15:10:07.811501 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.811484 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c"} err="failed to get container status \"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c\": rpc error: code = NotFound desc = could not find container \"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c\": container with ID starting with 0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c not found: ID does not exist" Nov 03 15:10:07.811501 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.811501 4557 scope.go:122] "RemoveContainer" containerID="87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a" Nov 03 15:10:07.811740 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.811717 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a"} err="failed to get container status \"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a\": rpc error: code = NotFound desc = could not find container \"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a\": container with ID starting with 87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a not found: ID does not exist" Nov 03 15:10:07.811740 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.811739 4557 scope.go:122] "RemoveContainer" containerID="3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025" Nov 03 15:10:07.811950 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.811932 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025"} err="failed to get container status \"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025\": rpc error: code = NotFound desc = could not find container \"3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025\": container with ID starting with 3be663f5d5f5097e6c133de0c80f43a868cd4b624f2defec428662894ac83025 not found: ID does not exist" Nov 03 15:10:07.812002 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.811950 4557 scope.go:122] "RemoveContainer" containerID="eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9" Nov 03 15:10:07.812186 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.812168 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9"} err="failed to get container status \"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9\": rpc error: code = NotFound desc = could not find container \"eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9\": container with ID starting with eb4e9d86c96faa887e247d724190abeb68e1737ccced3567058eaea0dc94a2e9 not found: ID does not exist" Nov 03 15:10:07.812226 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.812187 4557 scope.go:122] "RemoveContainer" containerID="12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37" Nov 03 15:10:07.812397 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.812376 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37"} err="failed to get container status \"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37\": rpc error: code = NotFound desc = could not find container \"12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37\": container with ID starting with 12e2b179f0546b7ed9fcbc5d167b4fa2ebda5e4e6ff857fb2aa582563f3efe37 not found: ID does not exist" Nov 03 15:10:07.812397 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.812397 4557 scope.go:122] "RemoveContainer" containerID="bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984" Nov 03 15:10:07.812615 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.812599 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984"} err="failed to get container status \"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984\": rpc error: code = NotFound desc = could not find container \"bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984\": container with ID starting with bd91ef2278f1558b7d5094ef169f9c76e7eb62a9a55de858bed1cde4590c2984 not found: ID does not exist" Nov 03 15:10:07.812669 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.812615 4557 scope.go:122] "RemoveContainer" containerID="f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713" Nov 03 15:10:07.812821 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.812804 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713"} err="failed to get container status \"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713\": rpc error: code = NotFound desc = could not find container \"f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713\": container with ID starting with f52ee8c8121e3271890b7d86297954a7b0246d6a56f95f71dfb115c3884ef713 not found: ID does not exist" Nov 03 15:10:07.812861 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.812822 4557 scope.go:122] "RemoveContainer" containerID="0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c" Nov 03 15:10:07.813009 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.812992 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c"} err="failed to get container status \"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c\": rpc error: code = NotFound desc = could not find container \"0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c\": container with ID starting with 0ad4c58e72cc8066be9023f54323a062543d7023fd1bfff3f15a07ea4ab1f13c not found: ID does not exist" Nov 03 15:10:07.813049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.813010 4557 scope.go:122] "RemoveContainer" containerID="87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a" Nov 03 15:10:07.813202 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.813185 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a"} err="failed to get container status \"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a\": rpc error: code = NotFound desc = could not find container \"87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a\": container with ID starting with 87d5389edd6e7d95ef6b6cdde47328e38fee3f1a7267773791aeca0a88a5db9a not found: ID does not exist" Nov 03 15:10:07.923893 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.923857 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 15:10:07.931643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.931619 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 15:10:07.964372 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964347 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 15:10:07.964569 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964557 4557 cpu_manager.go:414] "containerMap: removing container" podUID="3608eca4-f91b-4771-b523-1c5ea739c75a" containerName="console" Nov 03 15:10:07.964624 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964570 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3608eca4-f91b-4771-b523-1c5ea739c75a" containerName="console" Nov 03 15:10:07.964624 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964592 4557 cpu_manager.go:414] "containerMap: removing container" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="kube-rbac-proxy-thanos" Nov 03 15:10:07.964624 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964598 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="kube-rbac-proxy-thanos" Nov 03 15:10:07.964624 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964612 4557 cpu_manager.go:414] "containerMap: removing container" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="prometheus" Nov 03 15:10:07.964624 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964618 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="prometheus" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964629 4557 cpu_manager.go:414] "containerMap: removing container" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="init-config-reloader" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964634 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="init-config-reloader" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964639 4557 cpu_manager.go:414] "containerMap: removing container" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="kube-rbac-proxy-web" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964643 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="kube-rbac-proxy-web" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964649 4557 cpu_manager.go:414] "containerMap: removing container" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="kube-rbac-proxy" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964653 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="kube-rbac-proxy" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964660 4557 cpu_manager.go:414] "containerMap: removing container" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="thanos-sidecar" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964665 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="thanos-sidecar" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964674 4557 cpu_manager.go:414] "containerMap: removing container" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="config-reloader" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964679 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="config-reloader" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964684 4557 cpu_manager.go:414] "containerMap: removing container" podUID="450408b0-4d59-4e6d-a643-a2d1e15e459e" containerName="insights-gathering" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964688 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450408b0-4d59-4e6d-a643-a2d1e15e459e" containerName="insights-gathering" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964723 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="kube-rbac-proxy" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964730 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="prometheus" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964737 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="kube-rbac-proxy-web" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964744 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="thanos-sidecar" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964750 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="config-reloader" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964755 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="450408b0-4d59-4e6d-a643-a2d1e15e459e" containerName="insights-gathering" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964761 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="3608eca4-f91b-4771-b523-1c5ea739c75a" containerName="console" Nov 03 15:10:07.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.964767 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" containerName="kube-rbac-proxy-thanos" Nov 03 15:10:07.970038 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:07.970007 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.023398 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.023355 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 15:10:08.023667 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.023605 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 15:10:08.023667 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.023635 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 15:10:08.023900 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.023821 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 15:10:08.023900 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.023824 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 15:10:08.023900 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.023841 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 15:10:08.023900 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.023850 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 15:10:08.024076 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.023863 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 15:10:08.024076 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.023865 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 15:10:08.024076 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.023975 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 15:10:08.024076 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.023887 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 15:10:08.024076 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.023917 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hrfg2\"" Nov 03 15:10:08.024335 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.024100 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 15:10:08.024335 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.024117 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6oq4ppcs2gt9p\"" Nov 03 15:10:08.024335 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.024177 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 15:10:08.052042 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.051976 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.052042 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.052019 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-web-config\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.052290 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.052093 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.052290 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.052125 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.052290 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.052180 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.052290 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.052219 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.052526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.052341 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.052526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.052415 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-config-out\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.052526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.052444 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.052526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.052475 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.052774 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.052571 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.052774 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.052612 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-config\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.052774 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.052635 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.052774 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.052728 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6nql\" (UniqueName: \"kubernetes.io/projected/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-kube-api-access-x6nql\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.052774 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.052763 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.053072 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.052884 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.053072 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.052993 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.053072 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.053021 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.053072 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.053046 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.053222 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.053067 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.074277 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.074245 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 15:10:08.074428 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.074386 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 15:10:08.075308 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.075288 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 15:10:08.153930 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.153895 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.154137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.153938 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-config-out\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.154137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.153965 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.154137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.153993 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.154137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.154025 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.154137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.154045 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-config\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.154137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.154067 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.154137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.154089 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6nql\" (UniqueName: \"kubernetes.io/projected/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-kube-api-access-x6nql\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.154506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.154146 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.154506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.154203 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.154506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.154245 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.154506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.154268 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.154506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.154293 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.154506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.154316 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.154506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.154362 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.154506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.154387 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-web-config\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.154506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.154412 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.154506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.154444 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.154506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.154493 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.155101 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.154532 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.155101 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.154785 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.156089 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.155272 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.156089 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.155413 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.156089 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.155972 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.156813 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.156780 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.157178 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.157065 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.157178 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.157140 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.157567 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.157444 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.157567 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.157526 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.157759 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.157709 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-config-out\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.158222 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.158140 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.158222 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.158175 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.158386 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.158367 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.159190 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.159163 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.159301 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.159265 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.159424 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.159404 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.159718 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.159696 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-web-config\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.159867 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.159848 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-config\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.160377 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.160363 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.217362 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.217337 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6nql\" (UniqueName: \"kubernetes.io/projected/3c510ec8-7799-4f62-8e3b-7bdd94f95a2e-kube-api-access-x6nql\") pod \"prometheus-k8s-0\" (UID: \"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.282709 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.282675 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26b48fe9-4900-499f-b518-9a5f1150a0b1" path="/var/lib/kubelet/pods/26b48fe9-4900-499f-b518-9a5f1150a0b1/volumes" Nov 03 15:10:08.286121 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.286102 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:08.718485 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.718410 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 15:10:08.721188 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:10:08.721158 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c510ec8_7799_4f62_8e3b_7bdd94f95a2e.slice/crio-a2e0987019b2902867f07987b72fc30300e673782b4918e4e7014e9f563400eb WatchSource:0}: Error finding container a2e0987019b2902867f07987b72fc30300e673782b4918e4e7014e9f563400eb: Status 404 returned error can't find the container with id a2e0987019b2902867f07987b72fc30300e673782b4918e4e7014e9f563400eb Nov 03 15:10:08.759602 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:08.759555 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e","Type":"ContainerStarted","Data":"a2e0987019b2902867f07987b72fc30300e673782b4918e4e7014e9f563400eb"} Nov 03 15:10:09.765852 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:09.765814 4557 generic.go:356] "Generic (PLEG): container finished" podID="3c510ec8-7799-4f62-8e3b-7bdd94f95a2e" containerID="b3896cf057f8e45fcdddf1dc0c4a0de2ae88d1f5f88d7844c3a7ff1c97771b70" exitCode=0 Nov 03 15:10:09.766242 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:09.765862 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e","Type":"ContainerDied","Data":"b3896cf057f8e45fcdddf1dc0c4a0de2ae88d1f5f88d7844c3a7ff1c97771b70"} Nov 03 15:10:09.812157 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:09.812130 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:10:10.002968 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:10.002940 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:10:10.772934 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:10.772896 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e","Type":"ContainerStarted","Data":"5a9416e35afa3757098dedd69bf55db858182ebe18b76c5b7406340f330ace2d"} Nov 03 15:10:10.773308 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:10.772960 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e","Type":"ContainerStarted","Data":"34c8f8969737840a987c0516f911e934aed46d94b7a62637ac52ee944bc9a47e"} Nov 03 15:10:10.773308 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:10.772986 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e","Type":"ContainerStarted","Data":"16cbf519a73ca36830405bbb5d6bbe3ff092656e1b19b23a8f12dc967e880f4e"} Nov 03 15:10:10.773308 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:10.773010 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e","Type":"ContainerStarted","Data":"2ce9217975f7cfd8d036b9b8860c342c1f4e45efdcfc6c1358a4de44e4e0020a"} Nov 03 15:10:10.773308 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:10.773034 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e","Type":"ContainerStarted","Data":"2eaa5424f881badb300f4f7460c94f5fbd0b89c3ce558daaf78e0d4e68f571e2"} Nov 03 15:10:10.773308 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:10.773064 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c510ec8-7799-4f62-8e3b-7bdd94f95a2e","Type":"ContainerStarted","Data":"df4981242d970327900f87faf859742fcca28ce186a290ca7ffd389678264249"} Nov 03 15:10:10.906921 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:10.906870 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.8878651829999997 podStartE2EDuration="3.906857699s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.992516ms" isStatefulPod=false podCreationTimestamp="2026-11-03 15:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:10:10.906169046 +0000 UTC m=+120.749290611" watchObservedRunningTime="2026-11-03 15:10:10.906857699 +0000 UTC m=+120.749979240" Nov 03 15:10:13.287100 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:13.287061 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:10:16.529189 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:10:16.529152 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:10:46.529461 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:10:46.529375 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:10:55.762831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:10:55.762796 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:11:08.286952 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:11:08.286921 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:11:08.301888 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:11:08.301861 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:11:08.970185 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:11:08.970156 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 15:11:16.528642 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:11:16.528565 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:11:46.529404 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:11:46.529366 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:11:56.090021 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:11:56.089932 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:12:13.087471 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:12:13.087437 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:12:13.370965 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:12:13.370889 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:12:14.762690 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:12:14.762655 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:12:14.921975 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:12:14.921943 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:12:16.316313 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:12:16.316283 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:12:16.528119 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:12:16.528084 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:12:20.431602 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:12:20.431555 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:12:20.633666 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:12:20.633613 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:12:21.793824 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:12:21.793784 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:12:21.983808 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:12:21.983776 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:12:22.175232 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:12:22.175149 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:12:25.924759 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:12:25.924726 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:12:46.529161 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:12:46.529123 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:12:56.454951 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:12:56.454908 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:12:59.318998 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:12:59.318957 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:12:59.508294 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:12:59.508254 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:07.705836 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:07.705804 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:07.901630 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:07.901576 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:08.619599 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.619550 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:08.851596 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.851545 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr"] Nov 03 15:13:08.854986 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.854960 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:08.883528 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.883462 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8765c63a-431f-444a-94d9-d79f9f2020e8-tmp\") pod \"opendatahub-operator-controller-manager-84b7cd6487-ptqpr\" (UID: \"8765c63a-431f-444a-94d9-d79f9f2020e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:08.883528 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.883503 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8765c63a-431f-444a-94d9-d79f9f2020e8-metrics-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-ptqpr\" (UID: \"8765c63a-431f-444a-94d9-d79f9f2020e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:08.883528 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.883524 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8765c63a-431f-444a-94d9-d79f9f2020e8-manifests\") pod \"opendatahub-operator-controller-manager-84b7cd6487-ptqpr\" (UID: \"8765c63a-431f-444a-94d9-d79f9f2020e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:08.883856 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.883658 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8765c63a-431f-444a-94d9-d79f9f2020e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-ptqpr\" (UID: \"8765c63a-431f-444a-94d9-d79f9f2020e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:08.883856 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.883711 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8765c63a-431f-444a-94d9-d79f9f2020e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-ptqpr\" (UID: \"8765c63a-431f-444a-94d9-d79f9f2020e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:08.883856 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.883760 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqtf9\" (UniqueName: \"kubernetes.io/projected/8765c63a-431f-444a-94d9-d79f9f2020e8-kube-api-access-xqtf9\") pod \"opendatahub-operator-controller-manager-84b7cd6487-ptqpr\" (UID: \"8765c63a-431f-444a-94d9-d79f9f2020e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:08.901835 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.901808 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr"] Nov 03 15:13:08.912868 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.912831 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:08.949226 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.949195 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 15:13:08.949226 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.949198 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 15:13:08.949433 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.949391 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 15:13:08.949495 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.949436 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-nkgmk\"" Nov 03 15:13:08.949551 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.949541 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 15:13:08.949629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.949590 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 15:13:08.984977 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.984952 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8765c63a-431f-444a-94d9-d79f9f2020e8-tmp\") pod \"opendatahub-operator-controller-manager-84b7cd6487-ptqpr\" (UID: \"8765c63a-431f-444a-94d9-d79f9f2020e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:08.985111 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.984986 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8765c63a-431f-444a-94d9-d79f9f2020e8-metrics-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-ptqpr\" (UID: \"8765c63a-431f-444a-94d9-d79f9f2020e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:08.985111 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.985004 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8765c63a-431f-444a-94d9-d79f9f2020e8-manifests\") pod \"opendatahub-operator-controller-manager-84b7cd6487-ptqpr\" (UID: \"8765c63a-431f-444a-94d9-d79f9f2020e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:08.985111 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.985022 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8765c63a-431f-444a-94d9-d79f9f2020e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-ptqpr\" (UID: \"8765c63a-431f-444a-94d9-d79f9f2020e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:08.985111 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.985084 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8765c63a-431f-444a-94d9-d79f9f2020e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-ptqpr\" (UID: \"8765c63a-431f-444a-94d9-d79f9f2020e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:08.985284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.985114 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqtf9\" (UniqueName: \"kubernetes.io/projected/8765c63a-431f-444a-94d9-d79f9f2020e8-kube-api-access-xqtf9\") pod \"opendatahub-operator-controller-manager-84b7cd6487-ptqpr\" (UID: \"8765c63a-431f-444a-94d9-d79f9f2020e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:08.985469 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.985445 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8765c63a-431f-444a-94d9-d79f9f2020e8-manifests\") pod \"opendatahub-operator-controller-manager-84b7cd6487-ptqpr\" (UID: \"8765c63a-431f-444a-94d9-d79f9f2020e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:08.987324 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.987292 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8765c63a-431f-444a-94d9-d79f9f2020e8-tmp\") pod \"opendatahub-operator-controller-manager-84b7cd6487-ptqpr\" (UID: \"8765c63a-431f-444a-94d9-d79f9f2020e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:08.987428 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.987420 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8765c63a-431f-444a-94d9-d79f9f2020e8-metrics-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-ptqpr\" (UID: \"8765c63a-431f-444a-94d9-d79f9f2020e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:08.987497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.987421 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8765c63a-431f-444a-94d9-d79f9f2020e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-ptqpr\" (UID: \"8765c63a-431f-444a-94d9-d79f9f2020e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:08.987497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:08.987460 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8765c63a-431f-444a-94d9-d79f9f2020e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-ptqpr\" (UID: \"8765c63a-431f-444a-94d9-d79f9f2020e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:09.049879 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:09.049856 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqtf9\" (UniqueName: \"kubernetes.io/projected/8765c63a-431f-444a-94d9-d79f9f2020e8-kube-api-access-xqtf9\") pod \"opendatahub-operator-controller-manager-84b7cd6487-ptqpr\" (UID: \"8765c63a-431f-444a-94d9-d79f9f2020e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:09.103194 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:09.103157 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:09.169216 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:09.169155 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:09.298047 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:09.298011 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:09.663138 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:09.663114 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr"] Nov 03 15:13:09.664804 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:13:09.664778 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8765c63a_431f_444a_94d9_d79f9f2020e8.slice/crio-0c2658fc582d93e975a57a1ea2b0e39d139ba10f84c9326d517c1e06d8415990 WatchSource:0}: Error finding container 0c2658fc582d93e975a57a1ea2b0e39d139ba10f84c9326d517c1e06d8415990: Status 404 returned error can't find the container with id 0c2658fc582d93e975a57a1ea2b0e39d139ba10f84c9326d517c1e06d8415990 Nov 03 15:13:10.303056 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:10.303031 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:13:10.303695 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:10.303679 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:13:10.334310 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:10.334284 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" event={"ID":"8765c63a-431f-444a-94d9-d79f9f2020e8","Type":"ContainerStarted","Data":"0c2658fc582d93e975a57a1ea2b0e39d139ba10f84c9326d517c1e06d8415990"} Nov 03 15:13:10.622120 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:10.622034 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:10.879959 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:10.879861 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:11.910104 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:11.910059 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:12.703247 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:12.703188 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:12.896284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:12.896257 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:13.346335 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:13.346302 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" event={"ID":"8765c63a-431f-444a-94d9-d79f9f2020e8","Type":"ContainerStarted","Data":"de82fe34360acdf68a168b420d4d2da1105a5153b4d69858e6409d66b2191d07"} Nov 03 15:13:14.351445 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:14.351413 4557 generic.go:356] "Generic (PLEG): container finished" podID="8765c63a-431f-444a-94d9-d79f9f2020e8" containerID="de82fe34360acdf68a168b420d4d2da1105a5153b4d69858e6409d66b2191d07" exitCode=0 Nov 03 15:13:14.351871 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:14.351479 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" event={"ID":"8765c63a-431f-444a-94d9-d79f9f2020e8","Type":"ContainerDied","Data":"de82fe34360acdf68a168b420d4d2da1105a5153b4d69858e6409d66b2191d07"} Nov 03 15:13:14.352000 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:14.351985 4557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:13:15.356862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:15.356827 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" event={"ID":"8765c63a-431f-444a-94d9-d79f9f2020e8","Type":"ContainerStarted","Data":"3e66434d701f27ec383344cbd46a9fd5f9bce724149467b0a68bae55f70f71b3"} Nov 03 15:13:15.357258 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:15.356966 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:15.559440 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:15.559393 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" podStartSLOduration=3.03815862 podStartE2EDuration="7.559379757s" totalImagesPullingTime="4.112584881s" totalInitContainerRuntime="408.636256ms" isStatefulPod=false podCreationTimestamp="2026-11-03 15:13:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:15.556895503 +0000 UTC m=+305.400017044" watchObservedRunningTime="2026-11-03 15:13:15.559379757 +0000 UTC m=+305.402501296" Nov 03 15:13:16.528722 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:13:16.528682 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:13:17.741795 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:17.741757 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:17.932664 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:17.932628 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:21.508020 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:21.507972 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:24.378172 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:24.378095 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:26.364428 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:26.364394 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-ptqpr" Nov 03 15:13:28.591801 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:28.591758 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:28.843490 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:28.843410 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:30.762975 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:30.762939 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:30.958917 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:30.958885 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:31.427832 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:31.427802 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:31.655269 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:31.655240 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-2w8bg"] Nov 03 15:13:31.659565 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:31.659547 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-2w8bg" Nov 03 15:13:31.716665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:31.716596 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-2w8bg"] Nov 03 15:13:31.758231 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:31.758199 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn2wm\" (UniqueName: \"kubernetes.io/projected/2678f690-6a11-4419-a972-e09ab45ccace-kube-api-access-nn2wm\") pod \"cert-manager-cainjector-5c57bbd4fc-2w8bg\" (UID: \"2678f690-6a11-4419-a972-e09ab45ccace\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-2w8bg" Nov 03 15:13:31.758376 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:31.758274 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2678f690-6a11-4419-a972-e09ab45ccace-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-2w8bg\" (UID: \"2678f690-6a11-4419-a972-e09ab45ccace\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-2w8bg" Nov 03 15:13:31.762173 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:31.762153 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 15:13:31.762259 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:31.762182 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-dxgdb\"" Nov 03 15:13:31.762259 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:31.762193 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 15:13:31.859280 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:31.859252 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn2wm\" (UniqueName: \"kubernetes.io/projected/2678f690-6a11-4419-a972-e09ab45ccace-kube-api-access-nn2wm\") pod \"cert-manager-cainjector-5c57bbd4fc-2w8bg\" (UID: \"2678f690-6a11-4419-a972-e09ab45ccace\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-2w8bg" Nov 03 15:13:31.859280 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:31.859290 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2678f690-6a11-4419-a972-e09ab45ccace-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-2w8bg\" (UID: \"2678f690-6a11-4419-a972-e09ab45ccace\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-2w8bg" Nov 03 15:13:31.921136 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:31.921111 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2678f690-6a11-4419-a972-e09ab45ccace-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-2w8bg\" (UID: \"2678f690-6a11-4419-a972-e09ab45ccace\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-2w8bg" Nov 03 15:13:31.921269 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:31.921191 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn2wm\" (UniqueName: \"kubernetes.io/projected/2678f690-6a11-4419-a972-e09ab45ccace-kube-api-access-nn2wm\") pod \"cert-manager-cainjector-5c57bbd4fc-2w8bg\" (UID: \"2678f690-6a11-4419-a972-e09ab45ccace\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-2w8bg" Nov 03 15:13:31.983345 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:31.983267 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-2w8bg" Nov 03 15:13:32.349936 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:32.349906 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-2w8bg"] Nov 03 15:13:32.352822 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:13:32.352794 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2678f690_6a11_4419_a972_e09ab45ccace.slice/crio-0c01240d3381ee83c73cb974206a5ee52de5d96c762dc36b4e8b284acd1dcb18 WatchSource:0}: Error finding container 0c01240d3381ee83c73cb974206a5ee52de5d96c762dc36b4e8b284acd1dcb18: Status 404 returned error can't find the container with id 0c01240d3381ee83c73cb974206a5ee52de5d96c762dc36b4e8b284acd1dcb18 Nov 03 15:13:32.412291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:32.412264 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-2w8bg" event={"ID":"2678f690-6a11-4419-a972-e09ab45ccace","Type":"ContainerStarted","Data":"0c01240d3381ee83c73cb974206a5ee52de5d96c762dc36b4e8b284acd1dcb18"} Nov 03 15:13:33.882141 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:33.882100 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:34.684735 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:34.684699 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:35.232023 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:35.231988 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-tr6nx"] Nov 03 15:13:35.240436 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:35.240412 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tr6nx" Nov 03 15:13:35.286305 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:35.286269 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f27f30d-07d4-4ed3-b1db-4645289d94fe-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-tr6nx\" (UID: \"7f27f30d-07d4-4ed3-b1db-4645289d94fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tr6nx" Nov 03 15:13:35.286454 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:35.286400 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clgzs\" (UniqueName: \"kubernetes.io/projected/7f27f30d-07d4-4ed3-b1db-4645289d94fe-kube-api-access-clgzs\") pod \"cert-manager-webhook-5dbbbf75cc-tr6nx\" (UID: \"7f27f30d-07d4-4ed3-b1db-4645289d94fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tr6nx" Nov 03 15:13:35.292764 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:35.292729 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-tr6nx"] Nov 03 15:13:35.341629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:35.341595 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-zrfjk\"" Nov 03 15:13:35.387571 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:35.387525 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clgzs\" (UniqueName: \"kubernetes.io/projected/7f27f30d-07d4-4ed3-b1db-4645289d94fe-kube-api-access-clgzs\") pod \"cert-manager-webhook-5dbbbf75cc-tr6nx\" (UID: \"7f27f30d-07d4-4ed3-b1db-4645289d94fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tr6nx" Nov 03 15:13:35.387779 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:35.387668 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f27f30d-07d4-4ed3-b1db-4645289d94fe-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-tr6nx\" (UID: \"7f27f30d-07d4-4ed3-b1db-4645289d94fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tr6nx" Nov 03 15:13:35.449428 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:35.449397 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f27f30d-07d4-4ed3-b1db-4645289d94fe-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-tr6nx\" (UID: \"7f27f30d-07d4-4ed3-b1db-4645289d94fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tr6nx" Nov 03 15:13:35.449680 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:35.449649 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clgzs\" (UniqueName: \"kubernetes.io/projected/7f27f30d-07d4-4ed3-b1db-4645289d94fe-kube-api-access-clgzs\") pod \"cert-manager-webhook-5dbbbf75cc-tr6nx\" (UID: \"7f27f30d-07d4-4ed3-b1db-4645289d94fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tr6nx" Nov 03 15:13:35.488838 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:35.488768 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:35.553473 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:35.553426 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tr6nx" Nov 03 15:13:35.928300 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:35.928267 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-tr6nx"] Nov 03 15:13:35.932030 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:13:35.931995 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f27f30d_07d4_4ed3_b1db_4645289d94fe.slice/crio-69a7c5b283aafd592b870daeb94e9feed1c0290334c38ca42576f6015adb9b7e WatchSource:0}: Error finding container 69a7c5b283aafd592b870daeb94e9feed1c0290334c38ca42576f6015adb9b7e: Status 404 returned error can't find the container with id 69a7c5b283aafd592b870daeb94e9feed1c0290334c38ca42576f6015adb9b7e Nov 03 15:13:36.431569 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:36.431535 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tr6nx" event={"ID":"7f27f30d-07d4-4ed3-b1db-4645289d94fe","Type":"ContainerStarted","Data":"69a7c5b283aafd592b870daeb94e9feed1c0290334c38ca42576f6015adb9b7e"} Nov 03 15:13:37.436449 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:37.436415 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-2w8bg" event={"ID":"2678f690-6a11-4419-a972-e09ab45ccace","Type":"ContainerStarted","Data":"4e673d3ba8407579b2d5b28862c026a8040715d30f1db6dd91e01d25e03a5a89"} Nov 03 15:13:37.437966 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:37.437942 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tr6nx" event={"ID":"7f27f30d-07d4-4ed3-b1db-4645289d94fe","Type":"ContainerStarted","Data":"fc1d9c84dc4ab0f1eb5606eddd3b3cc31247a84cffa500b7d5e0d48cc8712e66"} Nov 03 15:13:37.438096 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:37.438053 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tr6nx" Nov 03 15:13:37.554187 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:37.554131 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-2w8bg" podStartSLOduration=1.686834988 podStartE2EDuration="6.554116753s" totalImagesPullingTime="4.867281765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:37.553852346 +0000 UTC m=+327.396973906" watchObservedRunningTime="2026-11-03 15:13:37.554116753 +0000 UTC m=+327.397238294" Nov 03 15:13:37.671351 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:37.671296 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tr6nx" podStartSLOduration=1.270908599 podStartE2EDuration="2.671282033s" totalImagesPullingTime="1.400373434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:37.670132603 +0000 UTC m=+327.513254146" watchObservedRunningTime="2026-11-03 15:13:37.671282033 +0000 UTC m=+327.514403574" Nov 03 15:13:38.819761 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:38.819729 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:39.023942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:39.023905 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:39.222865 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:39.222788 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:39.488685 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:39.488613 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:40.146926 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:40.146888 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:42.102284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:42.102247 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:42.805345 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:42.805311 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:43.444648 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:43.444621 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tr6nx" Nov 03 15:13:43.462230 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:43.462200 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-bcvlh"] Nov 03 15:13:43.465418 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:43.465401 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bcvlh" Nov 03 15:13:43.523664 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:43.523638 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 15:13:43.523664 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:43.523638 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-r82pz\"" Nov 03 15:13:43.547305 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:43.547282 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 15:13:43.547426 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:43.547314 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 15:13:43.552859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:43.552840 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgmmm\" (UniqueName: \"kubernetes.io/projected/7ea8b6c2-0d66-41bc-b592-6acb736f0246-kube-api-access-qgmmm\") pod \"dns-operator-controller-manager-8557cbdb84-bcvlh\" (UID: \"7ea8b6c2-0d66-41bc-b592-6acb736f0246\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bcvlh" Nov 03 15:13:43.562108 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:43.562091 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-bcvlh"] Nov 03 15:13:43.653548 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:43.653517 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgmmm\" (UniqueName: \"kubernetes.io/projected/7ea8b6c2-0d66-41bc-b592-6acb736f0246-kube-api-access-qgmmm\") pod \"dns-operator-controller-manager-8557cbdb84-bcvlh\" (UID: \"7ea8b6c2-0d66-41bc-b592-6acb736f0246\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bcvlh" Nov 03 15:13:43.716700 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:43.716631 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgmmm\" (UniqueName: \"kubernetes.io/projected/7ea8b6c2-0d66-41bc-b592-6acb736f0246-kube-api-access-qgmmm\") pod \"dns-operator-controller-manager-8557cbdb84-bcvlh\" (UID: \"7ea8b6c2-0d66-41bc-b592-6acb736f0246\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bcvlh" Nov 03 15:13:43.777058 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:43.777029 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bcvlh" Nov 03 15:13:44.099795 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:44.099763 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:44.149981 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:44.149951 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-bcvlh"] Nov 03 15:13:44.152811 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:13:44.152782 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea8b6c2_0d66_41bc_b592_6acb736f0246.slice/crio-ce66eb6e4ade6292221c0fec43a8c1fa92e10e5ba58bd07eee658b79a1d7e396 WatchSource:0}: Error finding container ce66eb6e4ade6292221c0fec43a8c1fa92e10e5ba58bd07eee658b79a1d7e396: Status 404 returned error can't find the container with id ce66eb6e4ade6292221c0fec43a8c1fa92e10e5ba58bd07eee658b79a1d7e396 Nov 03 15:13:44.461904 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:44.461815 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bcvlh" event={"ID":"7ea8b6c2-0d66-41bc-b592-6acb736f0246","Type":"ContainerStarted","Data":"ce66eb6e4ade6292221c0fec43a8c1fa92e10e5ba58bd07eee658b79a1d7e396"} Nov 03 15:13:44.733373 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:44.733291 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:45.668912 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:45.668882 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4c2w8"] Nov 03 15:13:45.673253 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:45.673236 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4c2w8" Nov 03 15:13:45.771629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:45.771576 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrkm8\" (UniqueName: \"kubernetes.io/projected/6b055afd-c279-404c-8554-4a7444bfb519-kube-api-access-mrkm8\") pod \"limitador-operator-controller-manager-5778bb4449-4c2w8\" (UID: \"6b055afd-c279-404c-8554-4a7444bfb519\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4c2w8" Nov 03 15:13:45.775928 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:45.775896 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xg5jz\"" Nov 03 15:13:45.777172 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:45.777151 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4c2w8"] Nov 03 15:13:45.872379 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:45.872341 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrkm8\" (UniqueName: \"kubernetes.io/projected/6b055afd-c279-404c-8554-4a7444bfb519-kube-api-access-mrkm8\") pod \"limitador-operator-controller-manager-5778bb4449-4c2w8\" (UID: \"6b055afd-c279-404c-8554-4a7444bfb519\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4c2w8" Nov 03 15:13:45.936188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:45.936125 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrkm8\" (UniqueName: \"kubernetes.io/projected/6b055afd-c279-404c-8554-4a7444bfb519-kube-api-access-mrkm8\") pod \"limitador-operator-controller-manager-5778bb4449-4c2w8\" (UID: \"6b055afd-c279-404c-8554-4a7444bfb519\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4c2w8" Nov 03 15:13:45.985901 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:45.985864 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4c2w8" Nov 03 15:13:46.289420 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:46.289346 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:46.433708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:46.433674 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4c2w8"] Nov 03 15:13:46.437109 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:13:46.437085 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b055afd_c279_404c_8554_4a7444bfb519.slice/crio-7024bf2f7ff89e0dba1cd95e832f3d99e91225fa112d1c100e4587405f558e8e WatchSource:0}: Error finding container 7024bf2f7ff89e0dba1cd95e832f3d99e91225fa112d1c100e4587405f558e8e: Status 404 returned error can't find the container with id 7024bf2f7ff89e0dba1cd95e832f3d99e91225fa112d1c100e4587405f558e8e Nov 03 15:13:46.470345 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:46.470316 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4c2w8" event={"ID":"6b055afd-c279-404c-8554-4a7444bfb519","Type":"ContainerStarted","Data":"7024bf2f7ff89e0dba1cd95e832f3d99e91225fa112d1c100e4587405f558e8e"} Nov 03 15:13:46.532232 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:13:46.532199 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:13:46.643905 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:46.643868 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:46.876305 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:46.876270 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:47.310914 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:47.310874 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:48.480629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:48.480575 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bcvlh" event={"ID":"7ea8b6c2-0d66-41bc-b592-6acb736f0246","Type":"ContainerStarted","Data":"5e81bac3334551278ac5364413a65602f441ecaab763482ba1159b3bb3cd2ff6"} Nov 03 15:13:48.481007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:48.480696 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bcvlh" Nov 03 15:13:48.661144 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:48.661093 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bcvlh" podStartSLOduration=2.180347709 podStartE2EDuration="5.661078837s" totalImagesPullingTime="3.480731128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:13:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:48.660357996 +0000 UTC m=+338.503479560" watchObservedRunningTime="2026-11-03 15:13:48.661078837 +0000 UTC m=+338.504200379" Nov 03 15:13:48.796853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:48.796818 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:48.994845 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:48.994809 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:49.150056 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.148013 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68fb7df4dc-28mfp"] Nov 03 15:13:49.152196 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.152172 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-28mfp" Nov 03 15:13:49.192853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.192824 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:49.205807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.205781 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 15:13:49.206049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.206035 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 15:13:49.206199 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.206188 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 15:13:49.206346 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.206334 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 15:13:49.206543 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.206526 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mzpz9\"" Nov 03 15:13:49.207178 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.207156 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68fb7df4dc-28mfp"] Nov 03 15:13:49.303434 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.303392 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4-tmp\") pod \"kube-auth-proxy-68fb7df4dc-28mfp\" (UID: \"809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4\") " pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-28mfp" Nov 03 15:13:49.303677 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.303450 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4-tls-certs\") pod \"kube-auth-proxy-68fb7df4dc-28mfp\" (UID: \"809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4\") " pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-28mfp" Nov 03 15:13:49.303677 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.303525 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q5nh\" (UniqueName: \"kubernetes.io/projected/809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4-kube-api-access-5q5nh\") pod \"kube-auth-proxy-68fb7df4dc-28mfp\" (UID: \"809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4\") " pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-28mfp" Nov 03 15:13:49.381496 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.381459 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:49.404645 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.404550 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q5nh\" (UniqueName: \"kubernetes.io/projected/809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4-kube-api-access-5q5nh\") pod \"kube-auth-proxy-68fb7df4dc-28mfp\" (UID: \"809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4\") " pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-28mfp" Nov 03 15:13:49.404827 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.404657 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4-tmp\") pod \"kube-auth-proxy-68fb7df4dc-28mfp\" (UID: \"809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4\") " pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-28mfp" Nov 03 15:13:49.404827 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.404705 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4-tls-certs\") pod \"kube-auth-proxy-68fb7df4dc-28mfp\" (UID: \"809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4\") " pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-28mfp" Nov 03 15:13:49.407245 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.407220 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4-tmp\") pod \"kube-auth-proxy-68fb7df4dc-28mfp\" (UID: \"809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4\") " pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-28mfp" Nov 03 15:13:49.407466 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.407442 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4-tls-certs\") pod \"kube-auth-proxy-68fb7df4dc-28mfp\" (UID: \"809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4\") " pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-28mfp" Nov 03 15:13:49.466653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.466616 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q5nh\" (UniqueName: \"kubernetes.io/projected/809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4-kube-api-access-5q5nh\") pod \"kube-auth-proxy-68fb7df4dc-28mfp\" (UID: \"809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4\") " pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-28mfp" Nov 03 15:13:49.467293 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.467275 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-28mfp" Nov 03 15:13:49.576115 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.576066 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:49.802252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.802168 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:49.862257 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.862222 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68fb7df4dc-28mfp"] Nov 03 15:13:49.865480 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:13:49.865454 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809d8f5b_51ed_4b00_a9fb_bd57ab23d5b4.slice/crio-569d94c065796d895538bf2bf2c124e91fd7e10f33f5a152522d2bc381bca0d5 WatchSource:0}: Error finding container 569d94c065796d895538bf2bf2c124e91fd7e10f33f5a152522d2bc381bca0d5: Status 404 returned error can't find the container with id 569d94c065796d895538bf2bf2c124e91fd7e10f33f5a152522d2bc381bca0d5 Nov 03 15:13:49.990868 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:49.990784 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:50.189843 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:50.189810 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:50.489017 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:50.488978 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-28mfp" event={"ID":"809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4","Type":"ContainerStarted","Data":"569d94c065796d895538bf2bf2c124e91fd7e10f33f5a152522d2bc381bca0d5"} Nov 03 15:13:50.490427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:50.490404 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4c2w8" event={"ID":"6b055afd-c279-404c-8554-4a7444bfb519","Type":"ContainerStarted","Data":"e8122d0cc5447cc6d92eebc704104b09892f301edfc2b86eae3186bfce4554a6"} Nov 03 15:13:50.490538 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:50.490531 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4c2w8" Nov 03 15:13:50.610501 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:50.610452 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4c2w8" podStartSLOduration=2.376178864 podStartE2EDuration="5.610438556s" totalImagesPullingTime="3.234259692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:13:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:50.609972696 +0000 UTC m=+340.453094238" watchObservedRunningTime="2026-11-03 15:13:50.610438556 +0000 UTC m=+340.453560097" Nov 03 15:13:51.029856 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:51.029818 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:51.223439 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:51.223396 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:53.504935 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:53.504901 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-28mfp" event={"ID":"809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4","Type":"ContainerStarted","Data":"0a719e41674aeb33a2589d7840411862a8f723796a192ab544157b63993a79a2"} Nov 03 15:13:53.622936 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:53.622884 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-28mfp" podStartSLOduration=1.728501231 podStartE2EDuration="4.622870465s" totalImagesPullingTime="2.894369234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:53.621205353 +0000 UTC m=+343.464326896" watchObservedRunningTime="2026-11-03 15:13:53.622870465 +0000 UTC m=+343.465992007" Nov 03 15:13:53.809035 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:53.809007 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mhr9s"] Nov 03 15:13:53.812001 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:53.811984 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mhr9s" Nov 03 15:13:53.864734 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:53.864709 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7rzzt\"" Nov 03 15:13:53.867065 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:53.867038 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mhr9s"] Nov 03 15:13:53.943805 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:53.943772 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9rp7\" (UniqueName: \"kubernetes.io/projected/97301630-5842-435c-92ad-130eecc5ac22-kube-api-access-z9rp7\") pod \"authorino-56c6c9ddd5-mhr9s\" (UID: \"97301630-5842-435c-92ad-130eecc5ac22\") " pod="kuadrant-system/authorino-56c6c9ddd5-mhr9s" Nov 03 15:13:54.044758 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:54.044729 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9rp7\" (UniqueName: \"kubernetes.io/projected/97301630-5842-435c-92ad-130eecc5ac22-kube-api-access-z9rp7\") pod \"authorino-56c6c9ddd5-mhr9s\" (UID: \"97301630-5842-435c-92ad-130eecc5ac22\") " pod="kuadrant-system/authorino-56c6c9ddd5-mhr9s" Nov 03 15:13:54.107235 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:54.107159 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9rp7\" (UniqueName: \"kubernetes.io/projected/97301630-5842-435c-92ad-130eecc5ac22-kube-api-access-z9rp7\") pod \"authorino-56c6c9ddd5-mhr9s\" (UID: \"97301630-5842-435c-92ad-130eecc5ac22\") " pod="kuadrant-system/authorino-56c6c9ddd5-mhr9s" Nov 03 15:13:54.123923 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:54.123888 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mhr9s" Nov 03 15:13:54.516091 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:54.516024 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mhr9s"] Nov 03 15:13:54.518908 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:13:54.518882 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97301630_5842_435c_92ad_130eecc5ac22.slice/crio-1282809f0402fcdc2d2087688184316867bf30869e49526f0f638fc78028b4b3 WatchSource:0}: Error finding container 1282809f0402fcdc2d2087688184316867bf30869e49526f0f638fc78028b4b3: Status 404 returned error can't find the container with id 1282809f0402fcdc2d2087688184316867bf30869e49526f0f638fc78028b4b3 Nov 03 15:13:55.513019 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:55.512983 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mhr9s" event={"ID":"97301630-5842-435c-92ad-130eecc5ac22","Type":"ContainerStarted","Data":"1282809f0402fcdc2d2087688184316867bf30869e49526f0f638fc78028b4b3"} Nov 03 15:13:56.800694 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:56.800659 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:56.989340 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:56.989305 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:57.521924 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:57.521891 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mhr9s" event={"ID":"97301630-5842-435c-92ad-130eecc5ac22","Type":"ContainerStarted","Data":"4a569e8948ad2bb6b0576785fb1395255987270e720cdd51093e3d7bf6d809ea"} Nov 03 15:13:57.638283 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:57.638225 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-mhr9s" podStartSLOduration=2.350797926 podStartE2EDuration="4.638212613s" totalImagesPullingTime="2.287414687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:57.637779631 +0000 UTC m=+347.480901174" watchObservedRunningTime="2026-11-03 15:13:57.638212613 +0000 UTC m=+347.481334149" Nov 03 15:13:58.944633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:58.944602 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:59.148812 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:59.148771 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:13:59.487440 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:13:59.487410 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bcvlh" Nov 03 15:14:01.386154 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:01.386114 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:01.497528 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:01.497498 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4c2w8" Nov 03 15:14:01.582974 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:01.582939 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:06.126845 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:06.126811 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:07.182962 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:07.182930 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:11.076412 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:11.076372 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:11.296973 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:11.296935 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:12.864357 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:12.864320 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:13.050954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:13.050919 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:13.256307 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:13.256225 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:15.086668 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:15.086639 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:15.301573 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:15.301541 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:16.528898 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:14:16.528857 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:14:19.888687 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:19.888653 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:20.089868 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:20.089833 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:22.330240 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:22.330206 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:22.540803 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:22.540765 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:26.841017 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:26.840985 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:26.888505 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:26.888475 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-w5hnw"] Nov 03 15:14:26.931875 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:26.931847 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-w5hnw" Nov 03 15:14:26.949773 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:26.949745 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-w5hnw"] Nov 03 15:14:26.998427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:26.998404 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-82fgg\"" Nov 03 15:14:26.998427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:26.998412 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 15:14:27.085234 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:27.085195 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:27.116345 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:27.116281 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0d6a17af-b654-4bf7-b683-158a9bd3f8ae-config-file\") pod \"limitador-limitador-74dfcfb894-w5hnw\" (UID: \"0d6a17af-b654-4bf7-b683-158a9bd3f8ae\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-w5hnw" Nov 03 15:14:27.116345 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:27.116340 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psxpv\" (UniqueName: \"kubernetes.io/projected/0d6a17af-b654-4bf7-b683-158a9bd3f8ae-kube-api-access-psxpv\") pod \"limitador-limitador-74dfcfb894-w5hnw\" (UID: \"0d6a17af-b654-4bf7-b683-158a9bd3f8ae\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-w5hnw" Nov 03 15:14:27.127346 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:27.127317 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-w5hnw"] Nov 03 15:14:27.217624 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:27.217597 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0d6a17af-b654-4bf7-b683-158a9bd3f8ae-config-file\") pod \"limitador-limitador-74dfcfb894-w5hnw\" (UID: \"0d6a17af-b654-4bf7-b683-158a9bd3f8ae\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-w5hnw" Nov 03 15:14:27.217771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:27.217672 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psxpv\" (UniqueName: \"kubernetes.io/projected/0d6a17af-b654-4bf7-b683-158a9bd3f8ae-kube-api-access-psxpv\") pod \"limitador-limitador-74dfcfb894-w5hnw\" (UID: \"0d6a17af-b654-4bf7-b683-158a9bd3f8ae\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-w5hnw" Nov 03 15:14:27.218199 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:27.218181 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0d6a17af-b654-4bf7-b683-158a9bd3f8ae-config-file\") pod \"limitador-limitador-74dfcfb894-w5hnw\" (UID: \"0d6a17af-b654-4bf7-b683-158a9bd3f8ae\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-w5hnw" Nov 03 15:14:27.273561 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:27.273529 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:27.279021 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:27.278997 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psxpv\" (UniqueName: \"kubernetes.io/projected/0d6a17af-b654-4bf7-b683-158a9bd3f8ae-kube-api-access-psxpv\") pod \"limitador-limitador-74dfcfb894-w5hnw\" (UID: \"0d6a17af-b654-4bf7-b683-158a9bd3f8ae\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-w5hnw" Nov 03 15:14:27.557943 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:27.557906 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-w5hnw" Nov 03 15:14:27.926520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:27.926443 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-w5hnw"] Nov 03 15:14:27.929508 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:14:27.929481 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6a17af_b654_4bf7_b683_158a9bd3f8ae.slice/crio-7e0fa8f9a092fb62a903aba4cd79b352d2e9b4e809f1b09e540cea3f74c8d257 WatchSource:0}: Error finding container 7e0fa8f9a092fb62a903aba4cd79b352d2e9b4e809f1b09e540cea3f74c8d257: Status 404 returned error can't find the container with id 7e0fa8f9a092fb62a903aba4cd79b352d2e9b4e809f1b09e540cea3f74c8d257 Nov 03 15:14:28.642698 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:28.642665 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-w5hnw" event={"ID":"0d6a17af-b654-4bf7-b683-158a9bd3f8ae","Type":"ContainerStarted","Data":"7e0fa8f9a092fb62a903aba4cd79b352d2e9b4e809f1b09e540cea3f74c8d257"} Nov 03 15:14:29.468393 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:29.468355 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-s5mfs"] Nov 03 15:14:29.471196 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:29.471176 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s5mfs" Nov 03 15:14:29.482769 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:29.482743 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:29.524110 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:29.524084 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 15:14:29.525759 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:29.525742 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 15:14:29.536005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:29.535978 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-s5mfs\" (UID: \"fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s5mfs" Nov 03 15:14:29.536139 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:29.536023 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvnhg\" (UniqueName: \"kubernetes.io/projected/fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0-kube-api-access-fvnhg\") pod \"kuadrant-console-plugin-66c4fb575d-s5mfs\" (UID: \"fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s5mfs" Nov 03 15:14:29.536139 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:29.536112 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-s5mfs\" (UID: \"fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s5mfs" Nov 03 15:14:29.574930 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:29.574899 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-s5mfs"] Nov 03 15:14:29.636962 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:29.636926 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-s5mfs\" (UID: \"fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s5mfs" Nov 03 15:14:29.637132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:29.637018 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-s5mfs\" (UID: \"fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s5mfs" Nov 03 15:14:29.637132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:29.637067 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvnhg\" (UniqueName: \"kubernetes.io/projected/fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0-kube-api-access-fvnhg\") pod \"kuadrant-console-plugin-66c4fb575d-s5mfs\" (UID: \"fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s5mfs" Nov 03 15:14:29.637252 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:14:29.637147 4557 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Nov 03 15:14:29.637252 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:14:29.637230 4557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0-plugin-serving-cert podName:fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0 nodeName:}" failed. No retries permitted until 2026-11-03 15:14:30.137207231 +0000 UTC m=+379.980328767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-s5mfs" (UID: "fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0") : secret "plugin-serving-cert" not found Nov 03 15:14:29.637684 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:29.637652 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-s5mfs\" (UID: \"fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s5mfs" Nov 03 15:14:29.701368 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:29.701338 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvnhg\" (UniqueName: \"kubernetes.io/projected/fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0-kube-api-access-fvnhg\") pod \"kuadrant-console-plugin-66c4fb575d-s5mfs\" (UID: \"fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s5mfs" Nov 03 15:14:29.820254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:29.820206 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:30.140811 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:30.140722 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-s5mfs\" (UID: \"fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s5mfs" Nov 03 15:14:30.142990 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:30.142972 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-s5mfs\" (UID: \"fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s5mfs" Nov 03 15:14:30.381209 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:30.381173 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s5mfs" Nov 03 15:14:30.652265 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:30.652235 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-w5hnw" event={"ID":"0d6a17af-b654-4bf7-b683-158a9bd3f8ae","Type":"ContainerStarted","Data":"0f24b991019705789c2c34a7f29f2e17783799056b98b76add609329812e41e4"} Nov 03 15:14:30.652748 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:30.652357 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-w5hnw" Nov 03 15:14:30.751324 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:14:30.751298 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb9b17e7_3939_4d8f_b7c3_29550c8ef8e0.slice/crio-76a042f8c5d67930ca13ea42e4f5af95e814cde532ec468fb336a422a2730381 WatchSource:0}: Error finding container 76a042f8c5d67930ca13ea42e4f5af95e814cde532ec468fb336a422a2730381: Status 404 returned error can't find the container with id 76a042f8c5d67930ca13ea42e4f5af95e814cde532ec468fb336a422a2730381 Nov 03 15:14:30.756183 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:30.756158 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-s5mfs"] Nov 03 15:14:30.769481 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:30.769433 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-w5hnw" podStartSLOduration=2.754708714 podStartE2EDuration="4.769420615s" totalImagesPullingTime="2.014711901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:14:30.769144388 +0000 UTC m=+380.612265926" watchObservedRunningTime="2026-11-03 15:14:30.769420615 +0000 UTC m=+380.612542156" Nov 03 15:14:31.657050 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:31.657014 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s5mfs" event={"ID":"fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0","Type":"ContainerStarted","Data":"76a042f8c5d67930ca13ea42e4f5af95e814cde532ec468fb336a422a2730381"} Nov 03 15:14:32.257558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:32.257522 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:32.450773 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:32.450741 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:32.665725 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:32.665654 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:32.857576 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:32.857539 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:33.122261 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:33.122232 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:36.678942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:36.678906 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s5mfs" event={"ID":"fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0","Type":"ContainerStarted","Data":"11ffc9e537e5d75fade29627c580858dbffde61901081848a3c29a8688c31100"} Nov 03 15:14:36.797195 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:36.797142 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s5mfs" podStartSLOduration=2.360590036 podStartE2EDuration="7.797128144s" totalImagesPullingTime="5.436538108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:14:36.796499501 +0000 UTC m=+386.639621043" watchObservedRunningTime="2026-11-03 15:14:36.797128144 +0000 UTC m=+386.640249724" Nov 03 15:14:41.266027 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:41.265992 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:41.452525 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:41.452449 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:41.658327 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:41.658290 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-w5hnw" Nov 03 15:14:43.722430 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:43.722399 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:43.912634 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:43.912600 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:44.945474 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:44.945441 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-w5hnw"] Nov 03 15:14:44.945879 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:44.945633 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-w5hnw" podUID="0d6a17af-b654-4bf7-b683-158a9bd3f8ae" containerName="limitador" containerID="cri-o://0f24b991019705789c2c34a7f29f2e17783799056b98b76add609329812e41e4" gracePeriod=30 Nov 03 15:14:45.487734 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:45.487707 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-w5hnw" Nov 03 15:14:45.565402 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:45.565379 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d6a17af-b654-4bf7-b683-158a9bd3f8ae-config-file\" (UniqueName: \"kubernetes.io/configmap/0d6a17af-b654-4bf7-b683-158a9bd3f8ae-config-file\") pod \"0d6a17af-b654-4bf7-b683-158a9bd3f8ae\" (UID: \"0d6a17af-b654-4bf7-b683-158a9bd3f8ae\") " Nov 03 15:14:45.565516 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:45.565414 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6a17af-b654-4bf7-b683-158a9bd3f8ae-kube-api-access-psxpv\" (UniqueName: \"kubernetes.io/projected/0d6a17af-b654-4bf7-b683-158a9bd3f8ae-kube-api-access-psxpv\") pod \"0d6a17af-b654-4bf7-b683-158a9bd3f8ae\" (UID: \"0d6a17af-b654-4bf7-b683-158a9bd3f8ae\") " Nov 03 15:14:45.565796 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:45.565768 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6a17af-b654-4bf7-b683-158a9bd3f8ae-config-file" pod "0d6a17af-b654-4bf7-b683-158a9bd3f8ae" (UID: "0d6a17af-b654-4bf7-b683-158a9bd3f8ae"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:14:45.567538 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:45.567519 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6a17af-b654-4bf7-b683-158a9bd3f8ae-kube-api-access-psxpv" pod "0d6a17af-b654-4bf7-b683-158a9bd3f8ae" (UID: "0d6a17af-b654-4bf7-b683-158a9bd3f8ae"). InnerVolumeSpecName "kube-api-access-psxpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:14:45.666076 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:45.666048 4557 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0d6a17af-b654-4bf7-b683-158a9bd3f8ae-config-file\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:14:45.666076 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:45.666075 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psxpv\" (UniqueName: \"kubernetes.io/projected/0d6a17af-b654-4bf7-b683-158a9bd3f8ae-kube-api-access-psxpv\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:14:45.714516 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:45.714488 4557 generic.go:356] "Generic (PLEG): container finished" podID="0d6a17af-b654-4bf7-b683-158a9bd3f8ae" containerID="0f24b991019705789c2c34a7f29f2e17783799056b98b76add609329812e41e4" exitCode=0 Nov 03 15:14:45.714682 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:45.714549 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-w5hnw" Nov 03 15:14:45.714682 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:45.714560 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-w5hnw" event={"ID":"0d6a17af-b654-4bf7-b683-158a9bd3f8ae","Type":"ContainerDied","Data":"0f24b991019705789c2c34a7f29f2e17783799056b98b76add609329812e41e4"} Nov 03 15:14:45.714682 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:45.714627 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-w5hnw" event={"ID":"0d6a17af-b654-4bf7-b683-158a9bd3f8ae","Type":"ContainerDied","Data":"7e0fa8f9a092fb62a903aba4cd79b352d2e9b4e809f1b09e540cea3f74c8d257"} Nov 03 15:14:45.714682 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:45.714649 4557 scope.go:122] "RemoveContainer" containerID="0f24b991019705789c2c34a7f29f2e17783799056b98b76add609329812e41e4" Nov 03 15:14:45.722788 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:45.722770 4557 scope.go:122] "RemoveContainer" containerID="0f24b991019705789c2c34a7f29f2e17783799056b98b76add609329812e41e4" Nov 03 15:14:45.723035 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:14:45.723016 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f24b991019705789c2c34a7f29f2e17783799056b98b76add609329812e41e4\": container with ID starting with 0f24b991019705789c2c34a7f29f2e17783799056b98b76add609329812e41e4 not found: ID does not exist" containerID="0f24b991019705789c2c34a7f29f2e17783799056b98b76add609329812e41e4" Nov 03 15:14:45.723082 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:45.723047 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f24b991019705789c2c34a7f29f2e17783799056b98b76add609329812e41e4"} err="failed to get container status \"0f24b991019705789c2c34a7f29f2e17783799056b98b76add609329812e41e4\": rpc error: code = NotFound desc = could not find container \"0f24b991019705789c2c34a7f29f2e17783799056b98b76add609329812e41e4\": container with ID starting with 0f24b991019705789c2c34a7f29f2e17783799056b98b76add609329812e41e4 not found: ID does not exist" Nov 03 15:14:45.879894 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:45.879833 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-w5hnw"] Nov 03 15:14:45.884838 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:45.884817 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-w5hnw"] Nov 03 15:14:46.283111 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:46.283034 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6a17af-b654-4bf7-b683-158a9bd3f8ae" path="/var/lib/kubelet/pods/0d6a17af-b654-4bf7-b683-158a9bd3f8ae/volumes" Nov 03 15:14:46.528326 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:14:46.528285 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:14:48.470986 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:48.470953 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:48.658336 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:48.658301 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:50.455031 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:50.454988 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:50.651490 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:50.651458 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:51.122630 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.122595 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-8f78l"] Nov 03 15:14:51.122918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.122905 4557 cpu_manager.go:414] "containerMap: removing container" podUID="0d6a17af-b654-4bf7-b683-158a9bd3f8ae" containerName="limitador" Nov 03 15:14:51.122959 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.122920 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6a17af-b654-4bf7-b683-158a9bd3f8ae" containerName="limitador" Nov 03 15:14:51.123033 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.122992 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6a17af-b654-4bf7-b683-158a9bd3f8ae" containerName="limitador" Nov 03 15:14:51.127150 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.127135 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8f78l" Nov 03 15:14:51.180760 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.180727 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 15:14:51.184995 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.184968 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-khwnc\"" Nov 03 15:14:51.185151 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.184978 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:14:51.211364 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.211328 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e0e3856c-467a-4e0a-bec1-0873dc8bb5c2-data\") pod \"postgres-7cdbf6dc44-8f78l\" (UID: \"e0e3856c-467a-4e0a-bec1-0873dc8bb5c2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8f78l" Nov 03 15:14:51.211472 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.211391 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcfdw\" (UniqueName: \"kubernetes.io/projected/e0e3856c-467a-4e0a-bec1-0873dc8bb5c2-kube-api-access-pcfdw\") pod \"postgres-7cdbf6dc44-8f78l\" (UID: \"e0e3856c-467a-4e0a-bec1-0873dc8bb5c2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8f78l" Nov 03 15:14:51.230532 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.230510 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:14:51.231843 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.231825 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-8f78l"] Nov 03 15:14:51.253516 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.253486 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:51.312256 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.312228 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e0e3856c-467a-4e0a-bec1-0873dc8bb5c2-data\") pod \"postgres-7cdbf6dc44-8f78l\" (UID: \"e0e3856c-467a-4e0a-bec1-0873dc8bb5c2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8f78l" Nov 03 15:14:51.312398 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.312281 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcfdw\" (UniqueName: \"kubernetes.io/projected/e0e3856c-467a-4e0a-bec1-0873dc8bb5c2-kube-api-access-pcfdw\") pod \"postgres-7cdbf6dc44-8f78l\" (UID: \"e0e3856c-467a-4e0a-bec1-0873dc8bb5c2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8f78l" Nov 03 15:14:51.312669 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.312647 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e0e3856c-467a-4e0a-bec1-0873dc8bb5c2-data\") pod \"postgres-7cdbf6dc44-8f78l\" (UID: \"e0e3856c-467a-4e0a-bec1-0873dc8bb5c2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8f78l" Nov 03 15:14:51.374570 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.374511 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcfdw\" (UniqueName: \"kubernetes.io/projected/e0e3856c-467a-4e0a-bec1-0873dc8bb5c2-kube-api-access-pcfdw\") pod \"postgres-7cdbf6dc44-8f78l\" (UID: \"e0e3856c-467a-4e0a-bec1-0873dc8bb5c2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8f78l" Nov 03 15:14:51.439544 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.439511 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8f78l" Nov 03 15:14:51.809043 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:51.809015 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-8f78l"] Nov 03 15:14:51.812106 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:14:51.812076 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e3856c_467a_4e0a_bec1_0873dc8bb5c2.slice/crio-930ba1d4b7236cee384fcf88be3d874069c0d3bf3ae735d623ea7ffe64c2a4a4 WatchSource:0}: Error finding container 930ba1d4b7236cee384fcf88be3d874069c0d3bf3ae735d623ea7ffe64c2a4a4: Status 404 returned error can't find the container with id 930ba1d4b7236cee384fcf88be3d874069c0d3bf3ae735d623ea7ffe64c2a4a4 Nov 03 15:14:52.744928 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:52.744850 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8f78l" event={"ID":"e0e3856c-467a-4e0a-bec1-0873dc8bb5c2","Type":"ContainerStarted","Data":"930ba1d4b7236cee384fcf88be3d874069c0d3bf3ae735d623ea7ffe64c2a4a4"} Nov 03 15:14:57.221124 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:57.221005 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:57.887888 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:57.887844 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:58.093416 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:58.093382 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:14:58.773314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:58.773227 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8f78l" event={"ID":"e0e3856c-467a-4e0a-bec1-0873dc8bb5c2","Type":"ContainerStarted","Data":"e239de6cbf8df4276b810d512e0f64f559558fb626dd28f4342ae1e6e0a4ffeb"} Nov 03 15:14:58.773711 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:58.773352 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8f78l" Nov 03 15:14:58.898040 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:14:58.897985 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8f78l" podStartSLOduration=1.255092127 podStartE2EDuration="7.897970733s" totalImagesPullingTime="6.642878606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:14:58.89730892 +0000 UTC m=+408.740430461" watchObservedRunningTime="2026-11-03 15:14:58.897970733 +0000 UTC m=+408.741092276" Nov 03 15:15:04.806030 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:04.806000 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-8f78l" Nov 03 15:15:04.898131 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:04.898096 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:07.555021 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:07.554983 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:07.757080 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:07.757043 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:07.945691 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:07.945564 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:08.150674 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.150646 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:08.346192 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.346160 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:08.478290 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.478256 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-6n4n9"] Nov 03 15:15:08.483142 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.483122 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-6n4n9" Nov 03 15:15:08.541035 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.540999 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 15:15:08.541035 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.540999 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 15:15:08.564593 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.564547 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkksv\" (UniqueName: \"kubernetes.io/projected/006c2584-8b0a-48d1-ad51-14839b7ab01f-kube-api-access-hkksv\") pod \"kserve-controller-manager-776cc66d47-6n4n9\" (UID: \"006c2584-8b0a-48d1-ad51-14839b7ab01f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6n4n9" Nov 03 15:15:08.564939 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.564619 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/006c2584-8b0a-48d1-ad51-14839b7ab01f-cert\") pod \"kserve-controller-manager-776cc66d47-6n4n9\" (UID: \"006c2584-8b0a-48d1-ad51-14839b7ab01f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6n4n9" Nov 03 15:15:08.590899 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.590872 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-dxmz2\"" Nov 03 15:15:08.590899 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.590873 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 15:15:08.592253 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.592225 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-6n4n9"] Nov 03 15:15:08.593170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.593152 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r"] Nov 03 15:15:08.597754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.597704 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r" Nov 03 15:15:08.608356 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.608335 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:08.642187 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.642159 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r"] Nov 03 15:15:08.651022 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.650987 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-7788x\"" Nov 03 15:15:08.651022 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.650987 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 15:15:08.666057 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.666022 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/006c2584-8b0a-48d1-ad51-14839b7ab01f-cert\") pod \"kserve-controller-manager-776cc66d47-6n4n9\" (UID: \"006c2584-8b0a-48d1-ad51-14839b7ab01f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6n4n9" Nov 03 15:15:08.666198 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.666086 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkksv\" (UniqueName: \"kubernetes.io/projected/006c2584-8b0a-48d1-ad51-14839b7ab01f-kube-api-access-hkksv\") pod \"kserve-controller-manager-776cc66d47-6n4n9\" (UID: \"006c2584-8b0a-48d1-ad51-14839b7ab01f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6n4n9" Nov 03 15:15:08.671250 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.671224 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/006c2584-8b0a-48d1-ad51-14839b7ab01f-cert\") pod \"kserve-controller-manager-776cc66d47-6n4n9\" (UID: \"006c2584-8b0a-48d1-ad51-14839b7ab01f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6n4n9" Nov 03 15:15:08.729217 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.729181 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkksv\" (UniqueName: \"kubernetes.io/projected/006c2584-8b0a-48d1-ad51-14839b7ab01f-kube-api-access-hkksv\") pod \"kserve-controller-manager-776cc66d47-6n4n9\" (UID: \"006c2584-8b0a-48d1-ad51-14839b7ab01f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6n4n9" Nov 03 15:15:08.766977 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.766936 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55471ac0-aaeb-414d-bf96-89d04f576dea-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-l6z2r\" (UID: \"55471ac0-aaeb-414d-bf96-89d04f576dea\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r" Nov 03 15:15:08.767164 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.767005 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7d76\" (UniqueName: \"kubernetes.io/projected/55471ac0-aaeb-414d-bf96-89d04f576dea-kube-api-access-j7d76\") pod \"llmisvc-controller-manager-869d5dcc5d-l6z2r\" (UID: \"55471ac0-aaeb-414d-bf96-89d04f576dea\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r" Nov 03 15:15:08.788348 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.788317 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-2k2m2"] Nov 03 15:15:08.795739 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.795710 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-2k2m2" Nov 03 15:15:08.795885 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.795784 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-6n4n9" Nov 03 15:15:08.817206 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.817178 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:08.851179 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.851099 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-d9p42\"" Nov 03 15:15:08.851393 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.851240 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 15:15:08.851505 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.851325 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 15:15:08.851505 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.851391 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-2k2m2"] Nov 03 15:15:08.868222 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.868186 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7d76\" (UniqueName: \"kubernetes.io/projected/55471ac0-aaeb-414d-bf96-89d04f576dea-kube-api-access-j7d76\") pod \"llmisvc-controller-manager-869d5dcc5d-l6z2r\" (UID: \"55471ac0-aaeb-414d-bf96-89d04f576dea\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r" Nov 03 15:15:08.868369 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.868291 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55471ac0-aaeb-414d-bf96-89d04f576dea-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-l6z2r\" (UID: \"55471ac0-aaeb-414d-bf96-89d04f576dea\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r" Nov 03 15:15:08.873553 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.873510 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55471ac0-aaeb-414d-bf96-89d04f576dea-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-l6z2r\" (UID: \"55471ac0-aaeb-414d-bf96-89d04f576dea\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r" Nov 03 15:15:08.930973 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.930942 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7d76\" (UniqueName: \"kubernetes.io/projected/55471ac0-aaeb-414d-bf96-89d04f576dea-kube-api-access-j7d76\") pod \"llmisvc-controller-manager-869d5dcc5d-l6z2r\" (UID: \"55471ac0-aaeb-414d-bf96-89d04f576dea\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r" Nov 03 15:15:08.969260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.969228 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e6b3b69-64db-4006-8f4a-e78bd1e8cc05-cert\") pod \"odh-model-controller-769875db56-2k2m2\" (UID: \"9e6b3b69-64db-4006-8f4a-e78bd1e8cc05\") " pod="opendatahub/odh-model-controller-769875db56-2k2m2" Nov 03 15:15:08.969426 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.969266 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-498fz\" (UniqueName: \"kubernetes.io/projected/9e6b3b69-64db-4006-8f4a-e78bd1e8cc05-kube-api-access-498fz\") pod \"odh-model-controller-769875db56-2k2m2\" (UID: \"9e6b3b69-64db-4006-8f4a-e78bd1e8cc05\") " pod="opendatahub/odh-model-controller-769875db56-2k2m2" Nov 03 15:15:08.969426 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:08.969303 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e6b3b69-64db-4006-8f4a-e78bd1e8cc05-metrics-certs\") pod \"odh-model-controller-769875db56-2k2m2\" (UID: \"9e6b3b69-64db-4006-8f4a-e78bd1e8cc05\") " pod="opendatahub/odh-model-controller-769875db56-2k2m2" Nov 03 15:15:09.070739 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:09.070707 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e6b3b69-64db-4006-8f4a-e78bd1e8cc05-cert\") pod \"odh-model-controller-769875db56-2k2m2\" (UID: \"9e6b3b69-64db-4006-8f4a-e78bd1e8cc05\") " pod="opendatahub/odh-model-controller-769875db56-2k2m2" Nov 03 15:15:09.070739 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:09.070743 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-498fz\" (UniqueName: \"kubernetes.io/projected/9e6b3b69-64db-4006-8f4a-e78bd1e8cc05-kube-api-access-498fz\") pod \"odh-model-controller-769875db56-2k2m2\" (UID: \"9e6b3b69-64db-4006-8f4a-e78bd1e8cc05\") " pod="opendatahub/odh-model-controller-769875db56-2k2m2" Nov 03 15:15:09.070922 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:09.070774 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e6b3b69-64db-4006-8f4a-e78bd1e8cc05-metrics-certs\") pod \"odh-model-controller-769875db56-2k2m2\" (UID: \"9e6b3b69-64db-4006-8f4a-e78bd1e8cc05\") " pod="opendatahub/odh-model-controller-769875db56-2k2m2" Nov 03 15:15:09.073137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:09.073113 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e6b3b69-64db-4006-8f4a-e78bd1e8cc05-cert\") pod \"odh-model-controller-769875db56-2k2m2\" (UID: \"9e6b3b69-64db-4006-8f4a-e78bd1e8cc05\") " pod="opendatahub/odh-model-controller-769875db56-2k2m2" Nov 03 15:15:09.073247 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:09.073159 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e6b3b69-64db-4006-8f4a-e78bd1e8cc05-metrics-certs\") pod \"odh-model-controller-769875db56-2k2m2\" (UID: \"9e6b3b69-64db-4006-8f4a-e78bd1e8cc05\") " pod="opendatahub/odh-model-controller-769875db56-2k2m2" Nov 03 15:15:09.132780 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:09.132718 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-498fz\" (UniqueName: \"kubernetes.io/projected/9e6b3b69-64db-4006-8f4a-e78bd1e8cc05-kube-api-access-498fz\") pod \"odh-model-controller-769875db56-2k2m2\" (UID: \"9e6b3b69-64db-4006-8f4a-e78bd1e8cc05\") " pod="opendatahub/odh-model-controller-769875db56-2k2m2" Nov 03 15:15:09.184085 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:09.184061 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-6n4n9"] Nov 03 15:15:09.186413 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:15:09.186387 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006c2584_8b0a_48d1_ad51_14839b7ab01f.slice/crio-b1c4dacddc67e95430191abd8182e829ed75cd2aece3ca601665d5c808a8e714 WatchSource:0}: Error finding container b1c4dacddc67e95430191abd8182e829ed75cd2aece3ca601665d5c808a8e714: Status 404 returned error can't find the container with id b1c4dacddc67e95430191abd8182e829ed75cd2aece3ca601665d5c808a8e714 Nov 03 15:15:09.212668 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:09.212640 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r" Nov 03 15:15:09.412341 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:09.412265 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-2k2m2" Nov 03 15:15:09.674056 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:09.674028 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r"] Nov 03 15:15:09.676427 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:15:09.676402 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod55471ac0_aaeb_414d_bf96_89d04f576dea.slice/crio-0ff3185dbb2c773b7344b33c92772008580e86eb006dd29f8f2e2cce3502bbf9 WatchSource:0}: Error finding container 0ff3185dbb2c773b7344b33c92772008580e86eb006dd29f8f2e2cce3502bbf9: Status 404 returned error can't find the container with id 0ff3185dbb2c773b7344b33c92772008580e86eb006dd29f8f2e2cce3502bbf9 Nov 03 15:15:09.802983 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:09.802927 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-2k2m2"] Nov 03 15:15:09.807204 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:15:09.807170 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6b3b69_64db_4006_8f4a_e78bd1e8cc05.slice/crio-9313f84652a45eac10b8ec8f26596b5adfb9f0d4e8b2fd0322ab648816b91f2a WatchSource:0}: Error finding container 9313f84652a45eac10b8ec8f26596b5adfb9f0d4e8b2fd0322ab648816b91f2a: Status 404 returned error can't find the container with id 9313f84652a45eac10b8ec8f26596b5adfb9f0d4e8b2fd0322ab648816b91f2a Nov 03 15:15:09.817752 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:09.817726 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-6n4n9" event={"ID":"006c2584-8b0a-48d1-ad51-14839b7ab01f","Type":"ContainerStarted","Data":"b1c4dacddc67e95430191abd8182e829ed75cd2aece3ca601665d5c808a8e714"} Nov 03 15:15:09.818889 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:09.818853 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-2k2m2" event={"ID":"9e6b3b69-64db-4006-8f4a-e78bd1e8cc05","Type":"ContainerStarted","Data":"9313f84652a45eac10b8ec8f26596b5adfb9f0d4e8b2fd0322ab648816b91f2a"} Nov 03 15:15:09.819941 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:09.819923 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r" event={"ID":"55471ac0-aaeb-414d-bf96-89d04f576dea","Type":"ContainerStarted","Data":"0ff3185dbb2c773b7344b33c92772008580e86eb006dd29f8f2e2cce3502bbf9"} Nov 03 15:15:09.876964 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:09.876933 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:12.320737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:12.320704 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:12.519406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:12.519333 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:12.726014 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:12.725981 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:12.840836 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:12.840802 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r" event={"ID":"55471ac0-aaeb-414d-bf96-89d04f576dea","Type":"ContainerStarted","Data":"d09344b9af6690edf9fd4ed1a354e84da82653f099225ae926bfd314ac303c70"} Nov 03 15:15:12.841024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:12.840907 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r" Nov 03 15:15:12.842560 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:12.842539 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-6n4n9" event={"ID":"006c2584-8b0a-48d1-ad51-14839b7ab01f","Type":"ContainerStarted","Data":"126c1c9261b8005150a5c714c71010f6b9384c8ff441efb4cb1c38129fa124c0"} Nov 03 15:15:12.842747 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:12.842731 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-6n4n9" Nov 03 15:15:12.924247 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:12.924212 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:13.017029 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:13.016965 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r" podStartSLOduration=2.109617819 podStartE2EDuration="5.016947691s" totalImagesPullingTime="2.907329872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:15:13.016216583 +0000 UTC m=+422.859338126" watchObservedRunningTime="2026-11-03 15:15:13.016947691 +0000 UTC m=+422.860069235" Nov 03 15:15:13.194405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:13.194305 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-6n4n9" podStartSLOduration=2.286014977 podStartE2EDuration="5.194290293s" totalImagesPullingTime="2.908275316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:15:13.193245676 +0000 UTC m=+423.036367217" watchObservedRunningTime="2026-11-03 15:15:13.194290293 +0000 UTC m=+423.037411835" Nov 03 15:15:13.848061 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:13.848022 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-2k2m2" event={"ID":"9e6b3b69-64db-4006-8f4a-e78bd1e8cc05","Type":"ContainerStarted","Data":"e867cc104def1ee751446e6028e2ded9916a0eb5f52cd59857815a732641c1f7"} Nov 03 15:15:13.848467 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:13.848401 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-2k2m2" Nov 03 15:15:14.024710 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:14.024661 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-2k2m2" podStartSLOduration=2.621895408 podStartE2EDuration="6.02464734s" totalImagesPullingTime="3.402751932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:15:14.024271465 +0000 UTC m=+423.867393011" watchObservedRunningTime="2026-11-03 15:15:14.02464734 +0000 UTC m=+423.867768882" Nov 03 15:15:16.451254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:16.451214 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:16.529251 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:15:16.529215 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:15:16.641723 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:16.641689 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:16.837552 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:16.837521 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:17.025703 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:17.025667 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:17.216130 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:17.216055 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:17.408001 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:17.407965 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:17.649038 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:17.649002 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:17.836927 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:17.836898 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:18.031545 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:18.031467 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:18.983751 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:18.983706 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:19.171203 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:19.171172 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:19.364592 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:19.364557 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:22.492444 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:22.492409 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:22.693806 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:22.693773 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:25.857538 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:25.857508 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-2k2m2" Nov 03 15:15:26.173943 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:26.173862 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:26.377973 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:26.377939 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:29.266730 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:29.266694 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:43.850303 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:43.850272 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r" Nov 03 15:15:43.853215 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:43.853197 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-6n4n9" Nov 03 15:15:46.528857 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:15:46.528823 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:15:56.311377 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:56.311338 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:56.818049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:56.818021 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:57.554359 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:57.554319 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:58.307743 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:58.307707 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:58.492016 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:58.491982 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:58.780673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:58.780565 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:15:58.969752 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:15:58.969719 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:16:11.874137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:11.874101 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:16:12.214392 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:12.214311 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:16:12.566805 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:12.566768 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68785c6758-vmr8x"] Nov 03 15:16:12.569123 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:12.569104 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68785c6758-vmr8x" Nov 03 15:16:12.602861 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:12.602825 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9mzt\" (UniqueName: \"kubernetes.io/projected/18f305ac-c130-4492-96f9-c3dc13a00864-kube-api-access-h9mzt\") pod \"authorino-68785c6758-vmr8x\" (UID: \"18f305ac-c130-4492-96f9-c3dc13a00864\") " pod="kuadrant-system/authorino-68785c6758-vmr8x" Nov 03 15:16:12.628600 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:12.628561 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68785c6758-vmr8x"] Nov 03 15:16:12.703249 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:12.703211 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9mzt\" (UniqueName: \"kubernetes.io/projected/18f305ac-c130-4492-96f9-c3dc13a00864-kube-api-access-h9mzt\") pod \"authorino-68785c6758-vmr8x\" (UID: \"18f305ac-c130-4492-96f9-c3dc13a00864\") " pod="kuadrant-system/authorino-68785c6758-vmr8x" Nov 03 15:16:12.765227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:12.765190 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9mzt\" (UniqueName: \"kubernetes.io/projected/18f305ac-c130-4492-96f9-c3dc13a00864-kube-api-access-h9mzt\") pod \"authorino-68785c6758-vmr8x\" (UID: \"18f305ac-c130-4492-96f9-c3dc13a00864\") " pod="kuadrant-system/authorino-68785c6758-vmr8x" Nov 03 15:16:12.879885 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:12.879788 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68785c6758-vmr8x" Nov 03 15:16:13.254344 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:13.254318 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68785c6758-vmr8x"] Nov 03 15:16:13.257147 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:16:13.257116 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18f305ac_c130_4492_96f9_c3dc13a00864.slice/crio-cdf0f7c3fe7a215e716dcd6997127c58ea886e2bf990772c113225ad1fea0a00 WatchSource:0}: Error finding container cdf0f7c3fe7a215e716dcd6997127c58ea886e2bf990772c113225ad1fea0a00: Status 404 returned error can't find the container with id cdf0f7c3fe7a215e716dcd6997127c58ea886e2bf990772c113225ad1fea0a00 Nov 03 15:16:14.098001 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:14.097964 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68785c6758-vmr8x" event={"ID":"18f305ac-c130-4492-96f9-c3dc13a00864","Type":"ContainerStarted","Data":"cdf0f7c3fe7a215e716dcd6997127c58ea886e2bf990772c113225ad1fea0a00"} Nov 03 15:16:16.529132 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:16:16.529089 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:16:16.683513 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:16.683475 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68785c6758-vmr8x"] Nov 03 15:16:18.118514 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:18.118459 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68785c6758-vmr8x" event={"ID":"18f305ac-c130-4492-96f9-c3dc13a00864","Type":"ContainerStarted","Data":"8004eaef6c97ea00aeb75fbc1f4ad30cfab38de65dd3f6bfdefc2cda9e9225e0"} Nov 03 15:16:18.119037 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:18.118607 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-68785c6758-vmr8x" podUID="18f305ac-c130-4492-96f9-c3dc13a00864" containerName="authorino" containerID="cri-o://8004eaef6c97ea00aeb75fbc1f4ad30cfab38de65dd3f6bfdefc2cda9e9225e0" gracePeriod=30 Nov 03 15:16:18.234284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:18.234230 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-68785c6758-vmr8x" podStartSLOduration=2.356459965 podStartE2EDuration="6.234214793s" totalImagesPullingTime="3.877754828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:16:18.23292413 +0000 UTC m=+488.076045673" watchObservedRunningTime="2026-11-03 15:16:18.234214793 +0000 UTC m=+488.077336334" Nov 03 15:16:18.371280 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:18.371220 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68785c6758-vmr8x" Nov 03 15:16:18.447994 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:18.447961 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18f305ac-c130-4492-96f9-c3dc13a00864-kube-api-access-h9mzt\" (UniqueName: \"kubernetes.io/projected/18f305ac-c130-4492-96f9-c3dc13a00864-kube-api-access-h9mzt\") pod \"18f305ac-c130-4492-96f9-c3dc13a00864\" (UID: \"18f305ac-c130-4492-96f9-c3dc13a00864\") " Nov 03 15:16:18.450025 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:18.450004 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18f305ac-c130-4492-96f9-c3dc13a00864-kube-api-access-h9mzt" pod "18f305ac-c130-4492-96f9-c3dc13a00864" (UID: "18f305ac-c130-4492-96f9-c3dc13a00864"). InnerVolumeSpecName "kube-api-access-h9mzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:16:18.548599 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:18.548554 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9mzt\" (UniqueName: \"kubernetes.io/projected/18f305ac-c130-4492-96f9-c3dc13a00864-kube-api-access-h9mzt\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:16:19.125256 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:19.125226 4557 generic.go:356] "Generic (PLEG): container finished" podID="18f305ac-c130-4492-96f9-c3dc13a00864" containerID="8004eaef6c97ea00aeb75fbc1f4ad30cfab38de65dd3f6bfdefc2cda9e9225e0" exitCode=0 Nov 03 15:16:19.125685 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:19.125280 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68785c6758-vmr8x" Nov 03 15:16:19.125685 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:19.125282 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68785c6758-vmr8x" event={"ID":"18f305ac-c130-4492-96f9-c3dc13a00864","Type":"ContainerDied","Data":"8004eaef6c97ea00aeb75fbc1f4ad30cfab38de65dd3f6bfdefc2cda9e9225e0"} Nov 03 15:16:19.125685 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:19.125393 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68785c6758-vmr8x" event={"ID":"18f305ac-c130-4492-96f9-c3dc13a00864","Type":"ContainerDied","Data":"cdf0f7c3fe7a215e716dcd6997127c58ea886e2bf990772c113225ad1fea0a00"} Nov 03 15:16:19.125685 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:19.125421 4557 scope.go:122] "RemoveContainer" containerID="8004eaef6c97ea00aeb75fbc1f4ad30cfab38de65dd3f6bfdefc2cda9e9225e0" Nov 03 15:16:19.134320 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:19.134302 4557 scope.go:122] "RemoveContainer" containerID="8004eaef6c97ea00aeb75fbc1f4ad30cfab38de65dd3f6bfdefc2cda9e9225e0" Nov 03 15:16:19.134627 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:16:19.134610 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8004eaef6c97ea00aeb75fbc1f4ad30cfab38de65dd3f6bfdefc2cda9e9225e0\": container with ID starting with 8004eaef6c97ea00aeb75fbc1f4ad30cfab38de65dd3f6bfdefc2cda9e9225e0 not found: ID does not exist" containerID="8004eaef6c97ea00aeb75fbc1f4ad30cfab38de65dd3f6bfdefc2cda9e9225e0" Nov 03 15:16:19.134698 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:19.134642 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8004eaef6c97ea00aeb75fbc1f4ad30cfab38de65dd3f6bfdefc2cda9e9225e0"} err="failed to get container status \"8004eaef6c97ea00aeb75fbc1f4ad30cfab38de65dd3f6bfdefc2cda9e9225e0\": rpc error: code = NotFound desc = could not find container \"8004eaef6c97ea00aeb75fbc1f4ad30cfab38de65dd3f6bfdefc2cda9e9225e0\": container with ID starting with 8004eaef6c97ea00aeb75fbc1f4ad30cfab38de65dd3f6bfdefc2cda9e9225e0 not found: ID does not exist" Nov 03 15:16:19.289109 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:19.289075 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68785c6758-vmr8x"] Nov 03 15:16:19.339067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:19.339033 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-68785c6758-vmr8x"] Nov 03 15:16:20.050344 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:20.050310 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mhr9s"] Nov 03 15:16:20.050525 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:20.050490 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-mhr9s" podUID="97301630-5842-435c-92ad-130eecc5ac22" containerName="authorino" containerID="cri-o://4a569e8948ad2bb6b0576785fb1395255987270e720cdd51093e3d7bf6d809ea" gracePeriod=30 Nov 03 15:16:20.285041 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:20.285011 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18f305ac-c130-4492-96f9-c3dc13a00864" path="/var/lib/kubelet/pods/18f305ac-c130-4492-96f9-c3dc13a00864/volumes" Nov 03 15:16:20.293354 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:20.293329 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mhr9s" Nov 03 15:16:20.368051 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:20.367960 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97301630-5842-435c-92ad-130eecc5ac22-kube-api-access-z9rp7\" (UniqueName: \"kubernetes.io/projected/97301630-5842-435c-92ad-130eecc5ac22-kube-api-access-z9rp7\") pod \"97301630-5842-435c-92ad-130eecc5ac22\" (UID: \"97301630-5842-435c-92ad-130eecc5ac22\") " Nov 03 15:16:20.370036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:20.370010 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97301630-5842-435c-92ad-130eecc5ac22-kube-api-access-z9rp7" pod "97301630-5842-435c-92ad-130eecc5ac22" (UID: "97301630-5842-435c-92ad-130eecc5ac22"). InnerVolumeSpecName "kube-api-access-z9rp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:16:20.468598 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:20.468553 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9rp7\" (UniqueName: \"kubernetes.io/projected/97301630-5842-435c-92ad-130eecc5ac22-kube-api-access-z9rp7\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:16:20.581768 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:20.581730 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:16:20.779011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:20.778923 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:16:21.026272 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.026236 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-xhpgx"] Nov 03 15:16:21.026565 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.026553 4557 cpu_manager.go:414] "containerMap: removing container" podUID="18f305ac-c130-4492-96f9-c3dc13a00864" containerName="authorino" Nov 03 15:16:21.026637 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.026566 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18f305ac-c130-4492-96f9-c3dc13a00864" containerName="authorino" Nov 03 15:16:21.026637 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.026591 4557 cpu_manager.go:414] "containerMap: removing container" podUID="97301630-5842-435c-92ad-130eecc5ac22" containerName="authorino" Nov 03 15:16:21.026637 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.026599 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97301630-5842-435c-92ad-130eecc5ac22" containerName="authorino" Nov 03 15:16:21.026728 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.026671 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="18f305ac-c130-4492-96f9-c3dc13a00864" containerName="authorino" Nov 03 15:16:21.026728 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.026680 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="97301630-5842-435c-92ad-130eecc5ac22" containerName="authorino" Nov 03 15:16:21.028367 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.028353 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xhpgx" Nov 03 15:16:21.075052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.075020 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbzb8\" (UniqueName: \"kubernetes.io/projected/9a00bc65-dafa-429d-ba69-4dd8b9c8cbac-kube-api-access-lbzb8\") pod \"llmisvc-controller-manager-6466c554d8-xhpgx\" (UID: \"9a00bc65-dafa-429d-ba69-4dd8b9c8cbac\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xhpgx" Nov 03 15:16:21.075219 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.075061 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a00bc65-dafa-429d-ba69-4dd8b9c8cbac-cert\") pod \"llmisvc-controller-manager-6466c554d8-xhpgx\" (UID: \"9a00bc65-dafa-429d-ba69-4dd8b9c8cbac\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xhpgx" Nov 03 15:16:21.090025 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.089990 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-xhpgx"] Nov 03 15:16:21.135361 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.135324 4557 generic.go:356] "Generic (PLEG): container finished" podID="97301630-5842-435c-92ad-130eecc5ac22" containerID="4a569e8948ad2bb6b0576785fb1395255987270e720cdd51093e3d7bf6d809ea" exitCode=0 Nov 03 15:16:21.135526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.135375 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mhr9s" Nov 03 15:16:21.135526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.135395 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mhr9s" event={"ID":"97301630-5842-435c-92ad-130eecc5ac22","Type":"ContainerDied","Data":"4a569e8948ad2bb6b0576785fb1395255987270e720cdd51093e3d7bf6d809ea"} Nov 03 15:16:21.135526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.135436 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mhr9s" event={"ID":"97301630-5842-435c-92ad-130eecc5ac22","Type":"ContainerDied","Data":"1282809f0402fcdc2d2087688184316867bf30869e49526f0f638fc78028b4b3"} Nov 03 15:16:21.135526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.135459 4557 scope.go:122] "RemoveContainer" containerID="4a569e8948ad2bb6b0576785fb1395255987270e720cdd51093e3d7bf6d809ea" Nov 03 15:16:21.143812 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.143790 4557 scope.go:122] "RemoveContainer" containerID="4a569e8948ad2bb6b0576785fb1395255987270e720cdd51093e3d7bf6d809ea" Nov 03 15:16:21.144139 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:16:21.144111 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a569e8948ad2bb6b0576785fb1395255987270e720cdd51093e3d7bf6d809ea\": container with ID starting with 4a569e8948ad2bb6b0576785fb1395255987270e720cdd51093e3d7bf6d809ea not found: ID does not exist" containerID="4a569e8948ad2bb6b0576785fb1395255987270e720cdd51093e3d7bf6d809ea" Nov 03 15:16:21.144195 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.144148 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a569e8948ad2bb6b0576785fb1395255987270e720cdd51093e3d7bf6d809ea"} err="failed to get container status \"4a569e8948ad2bb6b0576785fb1395255987270e720cdd51093e3d7bf6d809ea\": rpc error: code = NotFound desc = could not find container \"4a569e8948ad2bb6b0576785fb1395255987270e720cdd51093e3d7bf6d809ea\": container with ID starting with 4a569e8948ad2bb6b0576785fb1395255987270e720cdd51093e3d7bf6d809ea not found: ID does not exist" Nov 03 15:16:21.176287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.176243 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbzb8\" (UniqueName: \"kubernetes.io/projected/9a00bc65-dafa-429d-ba69-4dd8b9c8cbac-kube-api-access-lbzb8\") pod \"llmisvc-controller-manager-6466c554d8-xhpgx\" (UID: \"9a00bc65-dafa-429d-ba69-4dd8b9c8cbac\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xhpgx" Nov 03 15:16:21.176459 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.176296 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a00bc65-dafa-429d-ba69-4dd8b9c8cbac-cert\") pod \"llmisvc-controller-manager-6466c554d8-xhpgx\" (UID: \"9a00bc65-dafa-429d-ba69-4dd8b9c8cbac\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xhpgx" Nov 03 15:16:21.178786 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.178761 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a00bc65-dafa-429d-ba69-4dd8b9c8cbac-cert\") pod \"llmisvc-controller-manager-6466c554d8-xhpgx\" (UID: \"9a00bc65-dafa-429d-ba69-4dd8b9c8cbac\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xhpgx" Nov 03 15:16:21.247390 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.247351 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbzb8\" (UniqueName: \"kubernetes.io/projected/9a00bc65-dafa-429d-ba69-4dd8b9c8cbac-kube-api-access-lbzb8\") pod \"llmisvc-controller-manager-6466c554d8-xhpgx\" (UID: \"9a00bc65-dafa-429d-ba69-4dd8b9c8cbac\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xhpgx" Nov 03 15:16:21.255010 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.254977 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mhr9s"] Nov 03 15:16:21.296656 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.296624 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mhr9s"] Nov 03 15:16:21.339989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.339955 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xhpgx" Nov 03 15:16:21.716661 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:21.716636 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-xhpgx"] Nov 03 15:16:21.719156 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:16:21.719127 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a00bc65_dafa_429d_ba69_4dd8b9c8cbac.slice/crio-4f92d89f5a8bd1b28fad32d7c3e6dc0c79a43be00fe39128288e3a02f059a83d WatchSource:0}: Error finding container 4f92d89f5a8bd1b28fad32d7c3e6dc0c79a43be00fe39128288e3a02f059a83d: Status 404 returned error can't find the container with id 4f92d89f5a8bd1b28fad32d7c3e6dc0c79a43be00fe39128288e3a02f059a83d Nov 03 15:16:22.141918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:22.141876 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xhpgx" event={"ID":"9a00bc65-dafa-429d-ba69-4dd8b9c8cbac","Type":"ContainerStarted","Data":"4f92d89f5a8bd1b28fad32d7c3e6dc0c79a43be00fe39128288e3a02f059a83d"} Nov 03 15:16:22.283896 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:22.283855 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97301630-5842-435c-92ad-130eecc5ac22" path="/var/lib/kubelet/pods/97301630-5842-435c-92ad-130eecc5ac22/volumes" Nov 03 15:16:23.148090 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:23.148005 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xhpgx" event={"ID":"9a00bc65-dafa-429d-ba69-4dd8b9c8cbac","Type":"ContainerStarted","Data":"73f1422610650e084467a0e80487aea250f5272fce6b80ae5290215ccb57fdc6"} Nov 03 15:16:23.148518 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:23.148160 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xhpgx" Nov 03 15:16:23.266164 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:23.266105 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xhpgx" podStartSLOduration=2.514256754 podStartE2EDuration="3.266089122s" totalImagesPullingTime="751.832368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:16:23.265252068 +0000 UTC m=+493.108373610" watchObservedRunningTime="2026-11-03 15:16:23.266089122 +0000 UTC m=+493.109210664" Nov 03 15:16:46.528905 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:16:46.528872 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:16:54.155030 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:54.154999 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xhpgx" Nov 03 15:16:54.423991 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:54.423918 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r"] Nov 03 15:16:54.424158 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:54.424115 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r" podUID="55471ac0-aaeb-414d-bf96-89d04f576dea" containerName="manager" containerID="cri-o://d09344b9af6690edf9fd4ed1a354e84da82653f099225ae926bfd314ac303c70" gracePeriod=30 Nov 03 15:16:54.660885 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:54.660857 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r" Nov 03 15:16:54.743478 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:54.743409 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55471ac0-aaeb-414d-bf96-89d04f576dea-cert\" (UniqueName: \"kubernetes.io/secret/55471ac0-aaeb-414d-bf96-89d04f576dea-cert\") pod \"55471ac0-aaeb-414d-bf96-89d04f576dea\" (UID: \"55471ac0-aaeb-414d-bf96-89d04f576dea\") " Nov 03 15:16:54.743478 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:54.743478 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55471ac0-aaeb-414d-bf96-89d04f576dea-kube-api-access-j7d76\" (UniqueName: \"kubernetes.io/projected/55471ac0-aaeb-414d-bf96-89d04f576dea-kube-api-access-j7d76\") pod \"55471ac0-aaeb-414d-bf96-89d04f576dea\" (UID: \"55471ac0-aaeb-414d-bf96-89d04f576dea\") " Nov 03 15:16:54.745504 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:54.745469 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55471ac0-aaeb-414d-bf96-89d04f576dea-cert" pod "55471ac0-aaeb-414d-bf96-89d04f576dea" (UID: "55471ac0-aaeb-414d-bf96-89d04f576dea"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:16:54.745633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:54.745511 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55471ac0-aaeb-414d-bf96-89d04f576dea-kube-api-access-j7d76" pod "55471ac0-aaeb-414d-bf96-89d04f576dea" (UID: "55471ac0-aaeb-414d-bf96-89d04f576dea"). InnerVolumeSpecName "kube-api-access-j7d76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:16:54.844654 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:54.844624 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7d76\" (UniqueName: \"kubernetes.io/projected/55471ac0-aaeb-414d-bf96-89d04f576dea-kube-api-access-j7d76\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:16:54.844654 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:54.844648 4557 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55471ac0-aaeb-414d-bf96-89d04f576dea-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:16:55.256256 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:55.256224 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:16:55.283085 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:55.283055 4557 generic.go:356] "Generic (PLEG): container finished" podID="55471ac0-aaeb-414d-bf96-89d04f576dea" containerID="d09344b9af6690edf9fd4ed1a354e84da82653f099225ae926bfd314ac303c70" exitCode=0 Nov 03 15:16:55.283205 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:55.283096 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r" event={"ID":"55471ac0-aaeb-414d-bf96-89d04f576dea","Type":"ContainerDied","Data":"d09344b9af6690edf9fd4ed1a354e84da82653f099225ae926bfd314ac303c70"} Nov 03 15:16:55.283205 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:55.283151 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r" event={"ID":"55471ac0-aaeb-414d-bf96-89d04f576dea","Type":"ContainerDied","Data":"0ff3185dbb2c773b7344b33c92772008580e86eb006dd29f8f2e2cce3502bbf9"} Nov 03 15:16:55.283205 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:55.283108 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r" Nov 03 15:16:55.283205 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:55.283176 4557 scope.go:122] "RemoveContainer" containerID="d09344b9af6690edf9fd4ed1a354e84da82653f099225ae926bfd314ac303c70" Nov 03 15:16:55.291887 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:55.291870 4557 scope.go:122] "RemoveContainer" containerID="d09344b9af6690edf9fd4ed1a354e84da82653f099225ae926bfd314ac303c70" Nov 03 15:16:55.292165 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:16:55.292144 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d09344b9af6690edf9fd4ed1a354e84da82653f099225ae926bfd314ac303c70\": container with ID starting with d09344b9af6690edf9fd4ed1a354e84da82653f099225ae926bfd314ac303c70 not found: ID does not exist" containerID="d09344b9af6690edf9fd4ed1a354e84da82653f099225ae926bfd314ac303c70" Nov 03 15:16:55.292229 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:55.292177 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09344b9af6690edf9fd4ed1a354e84da82653f099225ae926bfd314ac303c70"} err="failed to get container status \"d09344b9af6690edf9fd4ed1a354e84da82653f099225ae926bfd314ac303c70\": rpc error: code = NotFound desc = could not find container \"d09344b9af6690edf9fd4ed1a354e84da82653f099225ae926bfd314ac303c70\": container with ID starting with d09344b9af6690edf9fd4ed1a354e84da82653f099225ae926bfd314ac303c70 not found: ID does not exist" Nov 03 15:16:55.448028 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:55.447992 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r"] Nov 03 15:16:55.450090 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:55.450066 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:16:55.502414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:55.502387 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l6z2r"] Nov 03 15:16:56.283523 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:56.283489 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55471ac0-aaeb-414d-bf96-89d04f576dea" path="/var/lib/kubelet/pods/55471ac0-aaeb-414d-bf96-89d04f576dea/volumes" Nov 03 15:16:57.881828 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:16:57.881797 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:17:16.361114 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:16.361079 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:17:16.528931 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:17:16.528896 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:17:20.296442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:20.296410 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:17:23.887807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:23.887777 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:17:28.560551 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:28.560515 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:17:28.819855 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:28.819768 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:17:31.482424 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:31.482368 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:17:31.678378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:31.678344 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:17:39.377254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:39.377218 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:17:42.398851 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:42.398787 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:17:46.529741 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:17:46.529638 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:17:46.671699 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:46.671668 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:17:51.230890 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:51.230858 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:17:54.320285 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.320207 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t"] Nov 03 15:17:54.320681 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.320520 4557 cpu_manager.go:414] "containerMap: removing container" podUID="55471ac0-aaeb-414d-bf96-89d04f576dea" containerName="manager" Nov 03 15:17:54.320681 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.320531 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55471ac0-aaeb-414d-bf96-89d04f576dea" containerName="manager" Nov 03 15:17:54.320681 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.320605 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="55471ac0-aaeb-414d-bf96-89d04f576dea" containerName="manager" Nov 03 15:17:54.323276 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.323259 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.403170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.403133 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8345cb5f-d59d-44a3-b858-5885db9bb03f-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.403170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.403172 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8345cb5f-d59d-44a3-b858-5885db9bb03f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.403370 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.403198 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8345cb5f-d59d-44a3-b858-5885db9bb03f-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.403370 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.403268 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8345cb5f-d59d-44a3-b858-5885db9bb03f-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.403370 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.403299 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8345cb5f-d59d-44a3-b858-5885db9bb03f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.403466 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.403371 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8345cb5f-d59d-44a3-b858-5885db9bb03f-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.403466 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.403417 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpx2m\" (UniqueName: \"kubernetes.io/projected/8345cb5f-d59d-44a3-b858-5885db9bb03f-kube-api-access-jpx2m\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.403527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.403466 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8345cb5f-d59d-44a3-b858-5885db9bb03f-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.403563 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.403546 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8345cb5f-d59d-44a3-b858-5885db9bb03f-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.425947 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.425906 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-n82lg\"" Nov 03 15:17:54.426127 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.425982 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 15:17:54.463218 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.463188 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t"] Nov 03 15:17:54.504066 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.504035 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8345cb5f-d59d-44a3-b858-5885db9bb03f-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.504219 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.504067 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8345cb5f-d59d-44a3-b858-5885db9bb03f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.504219 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.504105 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8345cb5f-d59d-44a3-b858-5885db9bb03f-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.504219 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.504131 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8345cb5f-d59d-44a3-b858-5885db9bb03f-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.504219 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.504163 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8345cb5f-d59d-44a3-b858-5885db9bb03f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.504219 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.504195 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8345cb5f-d59d-44a3-b858-5885db9bb03f-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.504468 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.504219 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpx2m\" (UniqueName: \"kubernetes.io/projected/8345cb5f-d59d-44a3-b858-5885db9bb03f-kube-api-access-jpx2m\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.504468 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.504265 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8345cb5f-d59d-44a3-b858-5885db9bb03f-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.504468 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.504324 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8345cb5f-d59d-44a3-b858-5885db9bb03f-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.504651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.504477 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8345cb5f-d59d-44a3-b858-5885db9bb03f-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.504651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.504494 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8345cb5f-d59d-44a3-b858-5885db9bb03f-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.504796 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.504774 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8345cb5f-d59d-44a3-b858-5885db9bb03f-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.504865 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.504805 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8345cb5f-d59d-44a3-b858-5885db9bb03f-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.505391 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.505365 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8345cb5f-d59d-44a3-b858-5885db9bb03f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.506502 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.506472 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8345cb5f-d59d-44a3-b858-5885db9bb03f-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.506633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.506602 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8345cb5f-d59d-44a3-b858-5885db9bb03f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.565383 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.565360 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8345cb5f-d59d-44a3-b858-5885db9bb03f-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.565516 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.565455 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpx2m\" (UniqueName: \"kubernetes.io/projected/8345cb5f-d59d-44a3-b858-5885db9bb03f-kube-api-access-jpx2m\") pod \"maas-default-gateway-openshift-default-596dc9545f-9x97t\" (UID: \"8345cb5f-d59d-44a3-b858-5885db9bb03f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:54.643050 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:54.642969 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:17:55.069124 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:55.069099 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t"] Nov 03 15:17:55.071601 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:17:55.071556 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8345cb5f_d59d_44a3_b858_5885db9bb03f.slice/crio-1370db5e1e4955f0dbac1a5ecac7b83625f9549e2a7fff7cae85b862dff6f09c WatchSource:0}: Error finding container 1370db5e1e4955f0dbac1a5ecac7b83625f9549e2a7fff7cae85b862dff6f09c: Status 404 returned error can't find the container with id 1370db5e1e4955f0dbac1a5ecac7b83625f9549e2a7fff7cae85b862dff6f09c Nov 03 15:17:55.531300 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:55.531218 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" event={"ID":"8345cb5f-d59d-44a3-b858-5885db9bb03f","Type":"ContainerStarted","Data":"1370db5e1e4955f0dbac1a5ecac7b83625f9549e2a7fff7cae85b862dff6f09c"} Nov 03 15:17:58.242717 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:17:58.242670 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:18:04.670012 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:04.669984 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:18:04.670311 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:04.670052 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:18:04.670311 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:04.670082 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:18:05.572185 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:05.572145 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" event={"ID":"8345cb5f-d59d-44a3-b858-5885db9bb03f","Type":"ContainerStarted","Data":"ab0c6dfba0022408fd34c6e2b604f9b683018f8f09752bf3220046051e766f58"} Nov 03 15:18:05.643486 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:05.643449 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:18:05.644907 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:05.644884 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9x97t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 03 15:18:05.644996 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:05.644932 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" podUID="8345cb5f-d59d-44a3-b858-5885db9bb03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 03 15:18:05.798318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:05.796183 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" podStartSLOduration=2.200712833 podStartE2EDuration="11.796164439s" totalImagesPullingTime="9.595451606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:18:05.742851008 +0000 UTC m=+595.585972550" watchObservedRunningTime="2026-11-03 15:18:05.796164439 +0000 UTC m=+595.639285983" Nov 03 15:18:06.643966 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:06.643932 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9x97t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 03 15:18:06.644152 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:06.643988 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" podUID="8345cb5f-d59d-44a3-b858-5885db9bb03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 03 15:18:07.643551 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:07.643507 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9x97t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 03 15:18:07.643995 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:07.643571 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" podUID="8345cb5f-d59d-44a3-b858-5885db9bb03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 03 15:18:08.643906 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:08.643869 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9x97t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 03 15:18:08.644300 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:08.643921 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" podUID="8345cb5f-d59d-44a3-b858-5885db9bb03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 03 15:18:09.643280 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:09.643234 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:18:09.643506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:09.643481 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9x97t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 03 15:18:09.643572 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:09.643553 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" podUID="8345cb5f-d59d-44a3-b858-5885db9bb03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 03 15:18:10.330980 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:10.330951 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:18:10.332210 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:10.332187 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:18:10.644407 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:10.644314 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9x97t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 03 15:18:10.644407 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:10.644378 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" podUID="8345cb5f-d59d-44a3-b858-5885db9bb03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 03 15:18:11.553562 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:11.553526 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:18:11.643664 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:11.643623 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9x97t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 03 15:18:11.643868 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:11.643672 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" podUID="8345cb5f-d59d-44a3-b858-5885db9bb03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 03 15:18:12.768496 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:12.768463 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:18:12.769714 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:12.769693 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" Nov 03 15:18:16.529747 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:18:16.529711 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:18:21.180426 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:21.180383 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:18:21.365227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:21.365194 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:18:24.342228 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.342195 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r"] Nov 03 15:18:24.351408 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.351384 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.373840 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.373811 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2bc03b-b695-479c-a782-8939e59f1000-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.373959 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.373845 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzlmz\" (UniqueName: \"kubernetes.io/projected/7a2bc03b-b695-479c-a782-8939e59f1000-kube-api-access-lzlmz\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.373959 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.373875 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2bc03b-b695-479c-a782-8939e59f1000-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.374058 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.373985 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2bc03b-b695-479c-a782-8939e59f1000-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.374058 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.374010 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2bc03b-b695-479c-a782-8939e59f1000-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.374058 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.374034 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2bc03b-b695-479c-a782-8939e59f1000-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.374212 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.374066 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2bc03b-b695-479c-a782-8939e59f1000-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.374212 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.374139 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2bc03b-b695-479c-a782-8939e59f1000-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.374305 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.374238 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2bc03b-b695-479c-a782-8939e59f1000-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.475022 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.474997 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2bc03b-b695-479c-a782-8939e59f1000-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.475155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.475039 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2bc03b-b695-479c-a782-8939e59f1000-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.475155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.475056 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2bc03b-b695-479c-a782-8939e59f1000-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.475155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.475072 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2bc03b-b695-479c-a782-8939e59f1000-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.475155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.475103 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2bc03b-b695-479c-a782-8939e59f1000-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.475155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.475119 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2bc03b-b695-479c-a782-8939e59f1000-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.475155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.475146 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2bc03b-b695-479c-a782-8939e59f1000-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.475451 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.475177 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2bc03b-b695-479c-a782-8939e59f1000-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.475451 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.475204 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzlmz\" (UniqueName: \"kubernetes.io/projected/7a2bc03b-b695-479c-a782-8939e59f1000-kube-api-access-lzlmz\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.475562 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.475472 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2bc03b-b695-479c-a782-8939e59f1000-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.475562 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.475533 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2bc03b-b695-479c-a782-8939e59f1000-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.475690 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.475669 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2bc03b-b695-479c-a782-8939e59f1000-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.475762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.475746 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2bc03b-b695-479c-a782-8939e59f1000-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.475993 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.475974 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2bc03b-b695-479c-a782-8939e59f1000-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.477458 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.477437 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2bc03b-b695-479c-a782-8939e59f1000-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.477542 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.477485 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2bc03b-b695-479c-a782-8939e59f1000-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.536555 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.536526 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2bc03b-b695-479c-a782-8939e59f1000-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.536867 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.536847 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzlmz\" (UniqueName: \"kubernetes.io/projected/7a2bc03b-b695-479c-a782-8939e59f1000-kube-api-access-lzlmz\") pod \"maas-default-gateway-openshift-default-596dc9545f-qxl5r\" (UID: \"7a2bc03b-b695-479c-a782-8939e59f1000\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:24.569975 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.569950 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r"] Nov 03 15:18:24.664632 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:24.664550 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:25.093753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:25.093614 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r"] Nov 03 15:18:25.095927 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:18:25.095897 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2bc03b_b695_479c_a782_8939e59f1000.slice/crio-9256e23b094975015de0f7b660fee267e4f3a94cea409f0396c6336b11820c31 WatchSource:0}: Error finding container 9256e23b094975015de0f7b660fee267e4f3a94cea409f0396c6336b11820c31: Status 404 returned error can't find the container with id 9256e23b094975015de0f7b660fee267e4f3a94cea409f0396c6336b11820c31 Nov 03 15:18:25.097883 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:25.097853 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:18:25.097964 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:25.097915 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:18:25.097964 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:25.097943 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:18:25.674747 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:25.674704 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" event={"ID":"7a2bc03b-b695-479c-a782-8939e59f1000","Type":"ContainerStarted","Data":"bfbdf568feb6e2b02641a6f24170a0b2b921578858abee3dc24950af113102aa"} Nov 03 15:18:25.675098 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:25.674771 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" event={"ID":"7a2bc03b-b695-479c-a782-8939e59f1000","Type":"ContainerStarted","Data":"9256e23b094975015de0f7b660fee267e4f3a94cea409f0396c6336b11820c31"} Nov 03 15:18:25.902633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:25.902548 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" podStartSLOduration=1.90251826 podStartE2EDuration="1.90251826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:18:25.902127228 +0000 UTC m=+615.745248772" watchObservedRunningTime="2026-11-03 15:18:25.90251826 +0000 UTC m=+615.745639801" Nov 03 15:18:26.665753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:26.665718 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:26.667240 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:26.667214 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-qxl5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 03 15:18:26.667365 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:26.667262 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" podUID="7a2bc03b-b695-479c-a782-8939e59f1000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 03 15:18:27.665448 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:27.665411 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-qxl5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 03 15:18:27.665946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:27.665473 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" podUID="7a2bc03b-b695-479c-a782-8939e59f1000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 03 15:18:28.665372 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:28.665336 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-qxl5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 03 15:18:28.665603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:28.665425 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" podUID="7a2bc03b-b695-479c-a782-8939e59f1000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 03 15:18:29.665932 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:29.665898 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-qxl5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 03 15:18:29.666332 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:29.665950 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" podUID="7a2bc03b-b695-479c-a782-8939e59f1000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 03 15:18:29.957494 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:29.957409 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:18:30.202426 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:30.200966 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:18:30.665804 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:30.665770 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-qxl5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 03 15:18:30.665984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:30.665818 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" podUID="7a2bc03b-b695-479c-a782-8939e59f1000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 03 15:18:31.665495 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:31.665455 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-qxl5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 03 15:18:31.665690 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:31.665511 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" podUID="7a2bc03b-b695-479c-a782-8939e59f1000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 03 15:18:33.310379 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:33.310348 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:33.310875 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:33.310619 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:33.311458 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:33.311438 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" Nov 03 15:18:46.529303 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:18:46.529257 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:18:50.419042 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:50.419011 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:18:50.609808 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:50.609776 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:18:55.810857 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:55.810823 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:18:56.331815 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:56.331786 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:18:57.935654 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:57.935623 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:18:58.134241 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:58.134203 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:18:58.591697 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:58.591663 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:18:58.785593 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:58.785539 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:18:59.704909 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:59.704872 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:18:59.894661 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:18:59.894628 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:16.528826 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:19:16.528782 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:19:25.210072 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:25.209994 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:26.702632 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.702591 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw"] Nov 03 15:19:26.706028 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.706006 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.813990 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.813960 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.814188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.813997 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7bb3b78-db67-4de5-9b60-10d26274a199-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.814188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.814043 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.814188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.814091 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97lzj\" (UniqueName: \"kubernetes.io/projected/f7bb3b78-db67-4de5-9b60-10d26274a199-kube-api-access-97lzj\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.814188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.814182 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.814388 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.814215 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.814388 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.814272 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.814388 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.814294 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.814388 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.814310 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.859596 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.859544 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e62ff3-openshift-default-dockercfg-w2vlc\"" Nov 03 15:19:26.868477 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.868452 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw"] Nov 03 15:19:26.914842 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.914815 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.914999 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.914846 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97lzj\" (UniqueName: \"kubernetes.io/projected/f7bb3b78-db67-4de5-9b60-10d26274a199-kube-api-access-97lzj\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.915069 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.914989 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.915069 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.915029 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.915069 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.915063 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.915222 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.915095 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.915222 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.915124 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.915375 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.915353 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.915453 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.915428 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7bb3b78-db67-4de5-9b60-10d26274a199-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.915508 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.915459 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.915508 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.915491 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.915623 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.915548 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.915715 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.915691 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.916092 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.916072 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7bb3b78-db67-4de5-9b60-10d26274a199-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.917458 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.917431 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:26.917574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:26.917551 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:27.021001 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.020929 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97lzj\" (UniqueName: \"kubernetes.io/projected/f7bb3b78-db67-4de5-9b60-10d26274a199-kube-api-access-97lzj\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:27.027707 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.027680 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:27.055562 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.055527 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:27.073912 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.073882 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx"] Nov 03 15:19:27.078858 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.078833 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.117704 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.117669 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.117857 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.117718 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.117857 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.117742 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.117944 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.117875 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.117994 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.117941 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.118061 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.118041 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.118165 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.118141 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.118203 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.118185 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwkml\" (UniqueName: \"kubernetes.io/projected/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-kube-api-access-pwkml\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.118298 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.118273 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.125780 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.125746 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx"] Nov 03 15:19:27.196839 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.196812 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f78fd6-openshift-default-dockercfg-cd4bc\"" Nov 03 15:19:27.219571 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.219539 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.219764 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.219572 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.219764 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.219629 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.219764 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.219675 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.219764 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.219722 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.219999 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.219775 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.219999 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.219803 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwkml\" (UniqueName: \"kubernetes.io/projected/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-kube-api-access-pwkml\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.219999 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.219844 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.219999 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.219906 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.220213 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.220106 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.220213 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.220167 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.220368 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.220345 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.220430 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.220411 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.220695 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.220675 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.221937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.221916 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.222161 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.222145 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.258120 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.258075 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:27.304856 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.304824 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.305054 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.305031 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwkml\" (UniqueName: \"kubernetes.io/projected/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-kube-api-access-pwkml\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.318847 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.318828 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:27.402605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.402329 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:27.458446 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.458383 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:27.659100 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.659005 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:27.754346 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.754311 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw"] Nov 03 15:19:27.763511 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:19:27.763484 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bb3b78_db67_4de5_9b60_10d26274a199.slice/crio-7294eda2670fe3554672ccf453ecef55239b0c5b06e63fd9e5bb1d5f21677b53 WatchSource:0}: Error finding container 7294eda2670fe3554672ccf453ecef55239b0c5b06e63fd9e5bb1d5f21677b53: Status 404 returned error can't find the container with id 7294eda2670fe3554672ccf453ecef55239b0c5b06e63fd9e5bb1d5f21677b53 Nov 03 15:19:27.765829 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.765798 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:19:27.765916 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.765859 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:19:27.765916 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.765891 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:19:27.862114 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.862083 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:27.870335 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.870310 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx"] Nov 03 15:19:27.871593 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:19:27.871557 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda589a74b_03fc_4a08_bd68_9ad4066f7ce1.slice/crio-19f541ddaa81305865907ab2cb11317b80ed67becc496d79dc966688eb0b77b3 WatchSource:0}: Error finding container 19f541ddaa81305865907ab2cb11317b80ed67becc496d79dc966688eb0b77b3: Status 404 returned error can't find the container with id 19f541ddaa81305865907ab2cb11317b80ed67becc496d79dc966688eb0b77b3 Nov 03 15:19:27.873994 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.873968 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:19:27.874069 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.874033 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:19:27.874107 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.874071 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:19:27.965929 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.965494 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" event={"ID":"f7bb3b78-db67-4de5-9b60-10d26274a199","Type":"ContainerStarted","Data":"da8a589ec74bd3921a6aad779ff7607ca10de1c21d0ce5fba5c484f9e57c13bc"} Nov 03 15:19:27.965929 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.965555 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" event={"ID":"f7bb3b78-db67-4de5-9b60-10d26274a199","Type":"ContainerStarted","Data":"7294eda2670fe3554672ccf453ecef55239b0c5b06e63fd9e5bb1d5f21677b53"} Nov 03 15:19:27.966943 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.966837 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" event={"ID":"a589a74b-03fc-4a08-bd68-9ad4066f7ce1","Type":"ContainerStarted","Data":"8171fe2f20c451c3c9918071e458960fac162f4fe9f0a0259199fc7f2bf88e8e"} Nov 03 15:19:27.966943 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:27.966877 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" event={"ID":"a589a74b-03fc-4a08-bd68-9ad4066f7ce1","Type":"ContainerStarted","Data":"19f541ddaa81305865907ab2cb11317b80ed67becc496d79dc966688eb0b77b3"} Nov 03 15:19:28.112986 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:28.112943 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:28.177279 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:28.177173 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" podStartSLOduration=2.177154379 podStartE2EDuration="2.177154379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:28.124246646 +0000 UTC m=+677.967368188" watchObservedRunningTime="2026-11-03 15:19:28.177154379 +0000 UTC m=+678.020275921" Nov 03 15:19:28.319156 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:28.319127 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:28.324048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:28.324026 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:28.354867 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:28.354826 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" podStartSLOduration=2.354814004 podStartE2EDuration="2.354814004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:28.353517171 +0000 UTC m=+678.196638715" watchObservedRunningTime="2026-11-03 15:19:28.354814004 +0000 UTC m=+678.197935544" Nov 03 15:19:28.403815 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:28.403617 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:28.408244 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:28.408224 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:28.972173 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:28.972086 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:28.972173 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:28.972142 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:28.972830 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:28.972812 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:19:28.972891 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:28.972867 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:19:30.364949 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:30.364917 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:30.554889 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:30.554854 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:31.312475 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:31.312447 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:31.557606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:31.557548 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:31.755743 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:31.755667 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:33.240260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:33.240229 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:33.446826 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:33.446797 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:33.692777 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:33.692741 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:33.885292 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:33.885258 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:34.080527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:34.080500 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:35.780850 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:35.780818 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:35.971526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:35.971494 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:37.147001 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:37.146969 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:37.341523 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:37.341493 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:37.532049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:37.531964 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:38.694531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:38.694497 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:38.881900 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:38.881868 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:40.562482 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:40.562447 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:40.824354 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:40.824274 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:41.013994 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:41.013962 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:41.223331 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:41.223250 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:41.432943 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:41.432907 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:41.832420 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:41.832389 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:42.039844 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:42.039816 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:42.552904 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:42.552875 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:42.742825 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:42.742787 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:42.943984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:42.943906 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:43.812489 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:43.812455 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:44.001621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:44.001594 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:46.528817 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:19:46.528783 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:19:48.045709 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:48.045677 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:48.239455 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:48.239419 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:48.433302 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:48.433223 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:53.295677 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:53.295643 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:53.493075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:53.493028 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:55.861662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:55.861621 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:56.066479 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:56.066449 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:58.409036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:58.409006 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:58.603794 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:58.603754 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:58.807553 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:58.807523 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:58.916291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:58.916259 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7"] Nov 03 15:19:58.920161 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:58.920129 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" Nov 03 15:19:58.976778 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:58.976735 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-e62ff3\"" Nov 03 15:19:58.982639 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:58.978542 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7"] Nov 03 15:19:58.983953 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:58.983257 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-e62ff3-dockercfg-6z8ln\"" Nov 03 15:19:59.008875 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:59.008841 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:59.026172 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:59.026138 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4drpg\" (UniqueName: \"kubernetes.io/projected/cf100201-cb1e-462e-878e-ff94690746d7-kube-api-access-4drpg\") pod \"payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7\" (UID: \"cf100201-cb1e-462e-878e-ff94690746d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" Nov 03 15:19:59.026327 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:59.026213 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf100201-cb1e-462e-878e-ff94690746d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7\" (UID: \"cf100201-cb1e-462e-878e-ff94690746d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" Nov 03 15:19:59.127151 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:59.127065 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf100201-cb1e-462e-878e-ff94690746d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7\" (UID: \"cf100201-cb1e-462e-878e-ff94690746d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" Nov 03 15:19:59.127151 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:59.127127 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drpg\" (UniqueName: \"kubernetes.io/projected/cf100201-cb1e-462e-878e-ff94690746d7-kube-api-access-4drpg\") pod \"payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7\" (UID: \"cf100201-cb1e-462e-878e-ff94690746d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" Nov 03 15:19:59.127862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:59.127836 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf100201-cb1e-462e-878e-ff94690746d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7\" (UID: \"cf100201-cb1e-462e-878e-ff94690746d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" Nov 03 15:19:59.197718 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:59.197679 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:19:59.202673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:59.202638 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4drpg\" (UniqueName: \"kubernetes.io/projected/cf100201-cb1e-462e-878e-ff94690746d7-kube-api-access-4drpg\") pod \"payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7\" (UID: \"cf100201-cb1e-462e-878e-ff94690746d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" Nov 03 15:19:59.236759 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:59.236721 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" Nov 03 15:19:59.637177 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:59.637152 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7"] Nov 03 15:19:59.638485 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:19:59.638461 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf100201_cb1e_462e_878e_ff94690746d7.slice/crio-6cf5baef47898a5d3d838f1e890ad67cfe18da387502760dda3f8536516c532a WatchSource:0}: Error finding container 6cf5baef47898a5d3d838f1e890ad67cfe18da387502760dda3f8536516c532a: Status 404 returned error can't find the container with id 6cf5baef47898a5d3d838f1e890ad67cfe18da387502760dda3f8536516c532a Nov 03 15:19:59.640258 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:19:59.640238 4557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:20:00.124025 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:00.123991 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" event={"ID":"cf100201-cb1e-462e-878e-ff94690746d7","Type":"ContainerStarted","Data":"6cf5baef47898a5d3d838f1e890ad67cfe18da387502760dda3f8536516c532a"} Nov 03 15:20:03.101365 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.101326 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:03.143214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.143178 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" event={"ID":"cf100201-cb1e-462e-878e-ff94690746d7","Type":"ContainerStarted","Data":"5f1886533aae05aaf35fe19bbc428c70c2101e18c9cf3a143f5e96c86f6cdf46"} Nov 03 15:20:03.143389 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.143318 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" Nov 03 15:20:03.266555 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.266457 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" podStartSLOduration=2.568929475 podStartE2EDuration="5.266442867s" totalImagesPullingTime="2.697513392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:20:03.265897631 +0000 UTC m=+713.109019186" watchObservedRunningTime="2026-11-03 15:20:03.266442867 +0000 UTC m=+713.109564409" Nov 03 15:20:03.609177 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.609146 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl"] Nov 03 15:20:03.622350 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.622325 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:20:03.668043 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.667998 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-metrics-certs\") pod \"maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl\" (UID: \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:20:03.674632 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.668135 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl\" (UID: \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:20:03.674632 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.668172 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djz2p\" (UniqueName: \"kubernetes.io/projected/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-kube-api-access-djz2p\") pod \"maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl\" (UID: \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:20:03.674632 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.668192 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl\" (UID: \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:20:03.674632 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.673997 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl"] Nov 03 15:20:03.724729 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.724686 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-f59d26-serving-cert\"" Nov 03 15:20:03.724944 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.724796 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p84nb\"" Nov 03 15:20:03.730916 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.730889 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 15:20:03.769189 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.769156 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl\" (UID: \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:20:03.769361 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.769203 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djz2p\" (UniqueName: \"kubernetes.io/projected/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-kube-api-access-djz2p\") pod \"maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl\" (UID: \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:20:03.769361 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.769225 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl\" (UID: \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:20:03.769361 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.769269 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-metrics-certs\") pod \"maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl\" (UID: \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:20:03.769526 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:20:03.769386 4557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-serving-cert: secret "maas-api-e2e-worker-w6-f59d26-serving-cert" not found Nov 03 15:20:03.769526 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:20:03.769465 4557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-maas-api-tls podName:ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d nodeName:}" failed. No retries permitted until 2026-11-03 15:20:04.269443302 +0000 UTC m=+714.112564840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-maas-api-tls") pod "maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" (UID: "ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d") : secret "maas-api-e2e-worker-w6-f59d26-serving-cert" not found Nov 03 15:20:03.769872 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.769854 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl\" (UID: \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:20:03.771533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.771512 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-metrics-certs\") pod \"maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl\" (UID: \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:20:03.839843 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.839805 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:03.856976 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:03.856948 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djz2p\" (UniqueName: \"kubernetes.io/projected/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-kube-api-access-djz2p\") pod \"maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl\" (UID: \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:20:04.036094 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:04.036002 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:04.149347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:04.149314 4557 generic.go:356] "Generic (PLEG): container finished" podID="a589a74b-03fc-4a08-bd68-9ad4066f7ce1" containerID="8171fe2f20c451c3c9918071e458960fac162f4fe9f0a0259199fc7f2bf88e8e" exitCode=0 Nov 03 15:20:04.149846 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:04.149386 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" event={"ID":"a589a74b-03fc-4a08-bd68-9ad4066f7ce1","Type":"ContainerDied","Data":"8171fe2f20c451c3c9918071e458960fac162f4fe9f0a0259199fc7f2bf88e8e"} Nov 03 15:20:04.150014 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:04.149999 4557 scope.go:122] "RemoveContainer" containerID="8171fe2f20c451c3c9918071e458960fac162f4fe9f0a0259199fc7f2bf88e8e" Nov 03 15:20:04.151091 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:04.151066 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:20:04.151173 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:04.151131 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:20:04.151173 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:04.151163 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:20:04.230866 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:04.230834 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:04.274226 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:04.274190 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl\" (UID: \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:20:04.276664 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:04.276638 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl\" (UID: \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:20:04.536738 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:04.536706 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:20:04.922316 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:04.922289 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl"] Nov 03 15:20:04.923203 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:20:04.923182 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6782e4_9efb_4d59_b7e3_c9bdfd66de6d.slice/crio-2b744d79b2a2cd63631b202c4c7d0cfaf93e375188cc8a2535c588656ad96b71 WatchSource:0}: Error finding container 2b744d79b2a2cd63631b202c4c7d0cfaf93e375188cc8a2535c588656ad96b71: Status 404 returned error can't find the container with id 2b744d79b2a2cd63631b202c4c7d0cfaf93e375188cc8a2535c588656ad96b71 Nov 03 15:20:05.157507 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:05.157472 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" event={"ID":"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d","Type":"ContainerStarted","Data":"2b744d79b2a2cd63631b202c4c7d0cfaf93e375188cc8a2535c588656ad96b71"} Nov 03 15:20:05.159275 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:05.159249 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" event={"ID":"a589a74b-03fc-4a08-bd68-9ad4066f7ce1","Type":"ContainerStarted","Data":"eb2aa0669f07bc636d3e348d49eb3be5adcb85fbabd84abcf306496776996860"} Nov 03 15:20:05.403932 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:05.403893 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:20:05.404134 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:05.404113 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 03 15:20:05.404177 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:05.404158 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" podUID="a589a74b-03fc-4a08-bd68-9ad4066f7ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 03 15:20:06.403406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:06.403371 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 03 15:20:06.404068 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:06.403424 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" podUID="a589a74b-03fc-4a08-bd68-9ad4066f7ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 03 15:20:07.404373 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:07.404334 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 03 15:20:07.404799 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:07.404397 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" podUID="a589a74b-03fc-4a08-bd68-9ad4066f7ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 03 15:20:08.403702 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:08.403671 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 03 15:20:08.403811 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:08.403734 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" podUID="a589a74b-03fc-4a08-bd68-9ad4066f7ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 03 15:20:08.874640 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:08.874601 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:09.096605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:09.096208 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:09.204027 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:09.203932 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" event={"ID":"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d","Type":"ContainerStarted","Data":"fb6db875c5ee2b1903f052eae96d7e71ac5eefb887a13c5caca145119d0875c2"} Nov 03 15:20:09.204180 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:09.204049 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:20:09.302640 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:09.302567 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:09.331099 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:09.331052 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" podStartSLOduration=2.8525273220000003 podStartE2EDuration="6.33103965s" totalImagesPullingTime="3.478512328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:20:09.329952012 +0000 UTC m=+719.173073554" watchObservedRunningTime="2026-11-03 15:20:09.33103965 +0000 UTC m=+719.174161192" Nov 03 15:20:09.404286 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:09.404248 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 03 15:20:09.404475 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:09.404312 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" podUID="a589a74b-03fc-4a08-bd68-9ad4066f7ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 03 15:20:10.403449 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:10.403413 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 03 15:20:10.403878 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:10.403465 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" podUID="a589a74b-03fc-4a08-bd68-9ad4066f7ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 03 15:20:11.407807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:11.407777 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:20:11.408240 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:11.408093 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:20:11.408781 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:11.408760 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:20:13.540767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:13.540729 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:13.924930 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:13.924894 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:14.152393 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:14.152362 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" Nov 03 15:20:15.215114 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:15.215087 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:20:16.529103 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:20:16.529072 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:20:18.847652 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:18.847612 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:19.040718 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:19.040669 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:22.104659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:22.100381 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:22.728877 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:22.728842 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:22.918417 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:22.918386 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:24.419261 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:24.419234 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:26.540744 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.540708 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz"] Nov 03 15:20:26.544626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.544603 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.597331 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.597292 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-tls-certs\") pod \"simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.597490 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.597353 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-home\") pod \"simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.597490 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.597461 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-tmp-dir\") pod \"simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.597572 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.597490 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-model-cache\") pod \"simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.597572 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.597527 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48nzz\" (UniqueName: \"kubernetes.io/projected/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-kube-api-access-48nzz\") pod \"simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.597736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.597699 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-dshm\") pod \"simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.653474 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.653442 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9855ff\"/\"simulator-w1-9855ff-kserve-self-signed-certs\"" Nov 03 15:20:26.653675 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.653509 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9855ff\"/\"default-dockercfg-9n95r\"" Nov 03 15:20:26.653675 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.653522 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9855ff\"/\"kube-root-ca.crt\"" Nov 03 15:20:26.653675 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.653655 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9855ff\"/\"openshift-service-ca.crt\"" Nov 03 15:20:26.655377 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.655359 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz"] Nov 03 15:20:26.698471 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.698433 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-tmp-dir\") pod \"simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.698657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.698486 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-model-cache\") pod \"simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.698657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.698571 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48nzz\" (UniqueName: \"kubernetes.io/projected/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-kube-api-access-48nzz\") pod \"simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.698657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.698625 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-dshm\") pod \"simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.698948 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.698702 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-tls-certs\") pod \"simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.698948 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.698736 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-home\") pod \"simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.699066 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.699008 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-tmp-dir\") pod \"simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.699243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.699215 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-home\") pod \"simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.699416 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.699333 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-model-cache\") pod \"simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.701557 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.701533 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-dshm\") pod \"simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.703019 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.703001 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-tls-certs\") pod \"simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.711101 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.711071 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:26.764595 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.764560 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48nzz\" (UniqueName: \"kubernetes.io/projected/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-kube-api-access-48nzz\") pod \"simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:26.857602 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:26.857558 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:27.237826 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:27.237796 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz"] Nov 03 15:20:27.237928 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:20:27.237837 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod236d85e1_9c70_4f9a_8c6c_e41901f7bdc1.slice/crio-edcf99eae7743d5011694d637312a74c6226bc20dc509ee6ad63e6abd14b42e7 WatchSource:0}: Error finding container edcf99eae7743d5011694d637312a74c6226bc20dc509ee6ad63e6abd14b42e7: Status 404 returned error can't find the container with id edcf99eae7743d5011694d637312a74c6226bc20dc509ee6ad63e6abd14b42e7 Nov 03 15:20:27.285281 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:27.285245 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" event={"ID":"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1","Type":"ContainerStarted","Data":"edcf99eae7743d5011694d637312a74c6226bc20dc509ee6ad63e6abd14b42e7"} Nov 03 15:20:28.293370 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:28.293337 4557 generic.go:356] "Generic (PLEG): container finished" podID="f7bb3b78-db67-4de5-9b60-10d26274a199" containerID="da8a589ec74bd3921a6aad779ff7607ca10de1c21d0ce5fba5c484f9e57c13bc" exitCode=0 Nov 03 15:20:28.294098 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:28.293406 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" event={"ID":"f7bb3b78-db67-4de5-9b60-10d26274a199","Type":"ContainerDied","Data":"da8a589ec74bd3921a6aad779ff7607ca10de1c21d0ce5fba5c484f9e57c13bc"} Nov 03 15:20:28.294098 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:28.293879 4557 scope.go:122] "RemoveContainer" containerID="da8a589ec74bd3921a6aad779ff7607ca10de1c21d0ce5fba5c484f9e57c13bc" Nov 03 15:20:28.295443 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:28.295387 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:20:28.295612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:28.295476 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:20:28.295612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:28.295521 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:20:28.972672 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:28.972646 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:20:29.308621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:29.308562 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" event={"ID":"f7bb3b78-db67-4de5-9b60-10d26274a199","Type":"ContainerStarted","Data":"adcdbe15e376a9ee90739e444376a9875ff4302f5555d358cc517d5aba4f5033"} Nov 03 15:20:29.320066 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:29.320041 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:20:29.321436 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:29.321405 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 03 15:20:29.321564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:29.321459 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" podUID="f7bb3b78-db67-4de5-9b60-10d26274a199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 03 15:20:29.553619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:29.553564 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:30.314450 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:30.314414 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" event={"ID":"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1","Type":"ContainerStarted","Data":"2e8e6c4272a708744a6f3a5768852a1b5f3a4a028cf885fe15fffd8d6daa2ae6"} Nov 03 15:20:30.320040 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:30.320004 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 03 15:20:30.320191 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:30.320049 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" podUID="f7bb3b78-db67-4de5-9b60-10d26274a199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 03 15:20:30.439526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:30.439478 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podStartSLOduration=2.059833329 podStartE2EDuration="4.439464169s" totalImagesPullingTime="2.37963084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:20:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:20:30.438369746 +0000 UTC m=+740.281491289" watchObservedRunningTime="2026-11-03 15:20:30.439464169 +0000 UTC m=+740.282585710" Nov 03 15:20:30.857679 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:30.857649 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:31.319322 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:31.319287 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 03 15:20:31.319755 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:31.319346 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" podUID="f7bb3b78-db67-4de5-9b60-10d26274a199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 03 15:20:31.355431 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:31.355401 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:32.055683 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:32.055648 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:32.318170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:32.318039 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:32.319708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:32.319686 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 03 15:20:32.320044 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:32.319732 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" podUID="f7bb3b78-db67-4de5-9b60-10d26274a199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 03 15:20:33.319860 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:33.319825 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 03 15:20:33.320286 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:33.319887 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" podUID="f7bb3b78-db67-4de5-9b60-10d26274a199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 03 15:20:34.320172 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:34.320128 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 03 15:20:34.320694 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:34.320183 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" podUID="f7bb3b78-db67-4de5-9b60-10d26274a199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 03 15:20:34.905973 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:34.905936 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:35.323512 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:35.323482 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:20:35.323963 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:35.323763 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:20:35.324342 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:35.324328 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:20:36.858254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:36.858216 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:36.871106 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:36.871078 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:37.357663 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:37.357634 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:20:40.601950 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:40.601920 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:40.868622 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:40.868490 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:41.234513 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:41.234389 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:41.439442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:41.439413 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:46.423229 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:46.423195 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:46.529941 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:20:46.529903 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:20:51.156704 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:51.156658 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:51.847598 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:51.847508 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:52.070458 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:52.070414 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:54.093887 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:54.093806 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:54.695889 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:54.695847 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:54.923487 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:54.923455 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:56.102040 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.102010 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59"] Nov 03 15:20:56.107628 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.107607 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.167094 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.167058 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59"] Nov 03 15:20:56.185705 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.185679 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-dshm\") pod \"simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.185859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.185722 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-model-cache\") pod \"simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.185859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.185745 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-tmp-dir\") pod \"simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.185859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.185811 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da69fa9-d199-4563-9c71-6627369d0ef9-tls-certs\") pod \"simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.186027 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.185873 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-home\") pod \"simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.186027 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.185924 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8htmp\" (UniqueName: \"kubernetes.io/projected/0da69fa9-d199-4563-9c71-6627369d0ef9-kube-api-access-8htmp\") pod \"simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.216717 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.216693 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-34e8c5\"/\"default-dockercfg-sqn5c\"" Nov 03 15:20:56.216852 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.216756 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-34e8c5\"/\"kube-root-ca.crt\"" Nov 03 15:20:56.216852 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.216764 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-34e8c5\"/\"simulator-w5-34e8c5-kserve-self-signed-certs\"" Nov 03 15:20:56.216852 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.216806 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-34e8c5\"/\"openshift-service-ca.crt\"" Nov 03 15:20:56.255965 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.255925 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:56.286394 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.286353 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-model-cache\") pod \"simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.286394 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.286396 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-tmp-dir\") pod \"simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.286673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.286423 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da69fa9-d199-4563-9c71-6627369d0ef9-tls-certs\") pod \"simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.286673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.286507 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-home\") pod \"simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.286673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.286542 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8htmp\" (UniqueName: \"kubernetes.io/projected/0da69fa9-d199-4563-9c71-6627369d0ef9-kube-api-access-8htmp\") pod \"simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.286673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.286650 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-dshm\") pod \"simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.286856 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.286835 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-tmp-dir\") pod \"simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.286856 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.286848 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-model-cache\") pod \"simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.287118 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.287099 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-home\") pod \"simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.288884 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.288864 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-dshm\") pod \"simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.288947 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.288931 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da69fa9-d199-4563-9c71-6627369d0ef9-tls-certs\") pod \"simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.351960 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.351932 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8htmp\" (UniqueName: \"kubernetes.io/projected/0da69fa9-d199-4563-9c71-6627369d0ef9-kube-api-access-8htmp\") pod \"simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.418351 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.418253 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:56.810498 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:56.810470 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59"] Nov 03 15:20:56.811286 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:20:56.811261 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0da69fa9_d199_4563_9c71_6627369d0ef9.slice/crio-457b1fd2fc6d13b5436df070b36ab6c38fa9db8910cd97fe74002a5a3643fa38 WatchSource:0}: Error finding container 457b1fd2fc6d13b5436df070b36ab6c38fa9db8910cd97fe74002a5a3643fa38: Status 404 returned error can't find the container with id 457b1fd2fc6d13b5436df070b36ab6c38fa9db8910cd97fe74002a5a3643fa38 Nov 03 15:20:57.409326 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:57.409289 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:20:57.436366 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:57.436329 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" event={"ID":"0da69fa9-d199-4563-9c71-6627369d0ef9","Type":"ContainerStarted","Data":"3dc73bccfbddbab4f758277cbb351920d7640a9e47d8a5cc57857bd6f29d70c3"} Nov 03 15:20:57.436551 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:57.436379 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" event={"ID":"0da69fa9-d199-4563-9c71-6627369d0ef9","Type":"ContainerStarted","Data":"457b1fd2fc6d13b5436df070b36ab6c38fa9db8910cd97fe74002a5a3643fa38"} Nov 03 15:20:57.559444 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:57.559390 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podStartSLOduration=1.559375829 podStartE2EDuration="1.559375829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:20:57.558019389 +0000 UTC m=+767.401140922" watchObservedRunningTime="2026-11-03 15:20:57.559375829 +0000 UTC m=+767.402497370" Nov 03 15:20:58.419199 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:58.419160 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:20:59.453793 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:20:59.453766 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:01.469353 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:01.469321 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:01.820042 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:01.820006 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w"] Nov 03 15:21:01.823564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:01.823546 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:01.880562 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:01.880530 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f78fd6\"/\"openshift-service-ca.crt\"" Nov 03 15:21:01.880766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:01.880572 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f78fd6\"/\"kube-root-ca.crt\"" Nov 03 15:21:01.932480 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:01.932442 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f78fd6\"/\"simulator-w2-f78fd6-kserve-self-signed-certs\"" Nov 03 15:21:01.933967 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:01.933946 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w"] Nov 03 15:21:01.936027 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:01.936009 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f78fd6\"/\"default-dockercfg-6sggv\"" Nov 03 15:21:01.939431 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:01.939410 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-model-cache\") pod \"simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:01.939496 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:01.939469 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-home\") pod \"simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:01.939496 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:01.939486 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-dshm\") pod \"simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:01.939674 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:01.939657 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-tmp-dir\") pod \"simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:01.939739 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:01.939709 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b43d3a39-a80b-4a80-b04b-0ac762251667-tls-certs\") pod \"simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:01.939781 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:01.939762 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt5ng\" (UniqueName: \"kubernetes.io/projected/b43d3a39-a80b-4a80-b04b-0ac762251667-kube-api-access-vt5ng\") pod \"simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:01.994530 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:01.994489 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:02.040872 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:02.040837 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b43d3a39-a80b-4a80-b04b-0ac762251667-tls-certs\") pod \"simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:02.040872 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:02.040878 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt5ng\" (UniqueName: \"kubernetes.io/projected/b43d3a39-a80b-4a80-b04b-0ac762251667-kube-api-access-vt5ng\") pod \"simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:02.041163 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:02.040905 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-model-cache\") pod \"simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:02.041163 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:02.041032 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-home\") pod \"simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:02.041163 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:02.041062 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-dshm\") pod \"simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:02.041163 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:02.041125 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-tmp-dir\") pod \"simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:02.041417 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:02.041397 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-model-cache\") pod \"simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:02.041458 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:02.041439 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-home\") pod \"simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:02.041503 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:02.041463 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-tmp-dir\") pod \"simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:02.043372 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:02.043344 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-dshm\") pod \"simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:02.043647 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:02.043631 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b43d3a39-a80b-4a80-b04b-0ac762251667-tls-certs\") pod \"simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:02.140006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:02.139929 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt5ng\" (UniqueName: \"kubernetes.io/projected/b43d3a39-a80b-4a80-b04b-0ac762251667-kube-api-access-vt5ng\") pod \"simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:02.229025 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:02.228985 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:02.435665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:02.435555 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:02.465284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:02.465250 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:02.814025 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:02.814001 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w"] Nov 03 15:21:02.815272 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:21:02.815245 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb43d3a39_a80b_4a80_b04b_0ac762251667.slice/crio-5992275e3041a0f2f80ea401638d264300df2f8eff4e15bdf6873e8c65d93eef WatchSource:0}: Error finding container 5992275e3041a0f2f80ea401638d264300df2f8eff4e15bdf6873e8c65d93eef: Status 404 returned error can't find the container with id 5992275e3041a0f2f80ea401638d264300df2f8eff4e15bdf6873e8c65d93eef Nov 03 15:21:02.931744 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:02.931701 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:03.466988 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:03.466953 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" event={"ID":"b43d3a39-a80b-4a80-b04b-0ac762251667","Type":"ContainerStarted","Data":"eff23587f09058e71fa970defacd189cdab48a47837afd6855772b2a7b1e0a15"} Nov 03 15:21:03.466988 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:03.466994 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" event={"ID":"b43d3a39-a80b-4a80-b04b-0ac762251667","Type":"ContainerStarted","Data":"5992275e3041a0f2f80ea401638d264300df2f8eff4e15bdf6873e8c65d93eef"} Nov 03 15:21:03.589191 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:03.589129 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podStartSLOduration=2.589113233 podStartE2EDuration="2.589113233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:21:03.587493951 +0000 UTC m=+773.430615518" watchObservedRunningTime="2026-11-03 15:21:03.589113233 +0000 UTC m=+773.432234775" Nov 03 15:21:04.436003 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:04.435960 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:06.419267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:06.419229 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:21:06.432101 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:06.432073 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:21:06.491800 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:06.491772 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:21:08.288238 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:08.288200 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:09.698115 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:09.698078 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:10.346795 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:10.346752 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:10.562222 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:10.562193 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:12.436597 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:12.436557 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:12.449031 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:12.448999 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:12.520281 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:12.520251 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:21:13.542470 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:13.542438 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:15.731279 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.731246 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4"] Nov 03 15:21:15.735078 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.735052 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:15.843830 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.843795 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bc1f07\"/\"simulator-w4-bc1f07-kserve-self-signed-certs\"" Nov 03 15:21:15.843830 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.843795 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bc1f07\"/\"default-dockercfg-44z2w\"" Nov 03 15:21:15.844041 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.843919 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bc1f07\"/\"kube-root-ca.crt\"" Nov 03 15:21:15.844041 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.843965 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bc1f07\"/\"openshift-service-ca.crt\"" Nov 03 15:21:15.845589 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.845562 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4"] Nov 03 15:21:15.858242 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.858219 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:15.867388 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.867357 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-tmp-dir\") pod \"simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:15.867511 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.867392 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-tls-certs\") pod \"simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:15.867511 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.867470 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-dshm\") pod \"simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:15.867924 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.867518 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-model-cache\") pod \"simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:15.867924 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.867661 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-home\") pod \"simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:15.867924 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.867698 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dhds\" (UniqueName: \"kubernetes.io/projected/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-kube-api-access-7dhds\") pod \"simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:15.969033 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.968942 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-dshm\") pod \"simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:15.969189 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.969076 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-model-cache\") pod \"simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:15.969231 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.969205 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-home\") pod \"simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:15.969263 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.969231 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dhds\" (UniqueName: \"kubernetes.io/projected/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-kube-api-access-7dhds\") pod \"simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:15.969311 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.969270 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-tmp-dir\") pod \"simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:15.969311 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.969297 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-tls-certs\") pod \"simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:15.969571 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.969509 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-model-cache\") pod \"simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:15.969571 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.969551 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-home\") pod \"simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:15.969766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.969660 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-tmp-dir\") pod \"simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:15.971295 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.971270 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-dshm\") pod \"simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:15.971812 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:15.971792 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-tls-certs\") pod \"simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:16.035787 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:16.035716 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dhds\" (UniqueName: \"kubernetes.io/projected/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-kube-api-access-7dhds\") pod \"simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:16.046574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:16.046545 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:16.427994 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:16.427964 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4"] Nov 03 15:21:16.428456 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:21:16.428432 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd48d3ec_6e5d_492f_8651_7b0bdf52c3cc.slice/crio-c207583988b9a48162f523b469d9f808ec2b1808a74f8f6b40d180ed92df61f0 WatchSource:0}: Error finding container c207583988b9a48162f523b469d9f808ec2b1808a74f8f6b40d180ed92df61f0: Status 404 returned error can't find the container with id c207583988b9a48162f523b469d9f808ec2b1808a74f8f6b40d180ed92df61f0 Nov 03 15:21:16.530104 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:16.530069 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" event={"ID":"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc","Type":"ContainerStarted","Data":"60eeb37eb73bd7b6985faad3d28f43ebb6f134f942a68c1e2154943c4fc04308"} Nov 03 15:21:16.530258 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:16.530120 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" event={"ID":"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc","Type":"ContainerStarted","Data":"c207583988b9a48162f523b469d9f808ec2b1808a74f8f6b40d180ed92df61f0"} Nov 03 15:21:16.530635 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:21:16.530606 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:21:16.653508 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:16.653459 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podStartSLOduration=1.653447174 podStartE2EDuration="1.653447174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:21:16.652309158 +0000 UTC m=+786.495430712" watchObservedRunningTime="2026-11-03 15:21:16.653447174 +0000 UTC m=+786.496568715" Nov 03 15:21:17.047915 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:17.047865 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:18.467815 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:18.467762 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:23.413839 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:23.413786 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:26.047946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:26.047908 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:26.060559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:26.060529 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:26.585695 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:26.585665 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:21:28.405900 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:28.405866 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:29.387256 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.387220 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv"] Nov 03 15:21:29.390635 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.390619 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.450458 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.450427 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv"] Nov 03 15:21:29.498164 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.498128 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c821d166-2220-42fc-801d-62207ca877f2-tls-certs\") pod \"simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.498352 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.498182 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-model-cache\") pod \"simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.498352 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.498248 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-home\") pod \"simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.498352 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.498332 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnnzz\" (UniqueName: \"kubernetes.io/projected/c821d166-2220-42fc-801d-62207ca877f2-kube-api-access-fnnzz\") pod \"simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.498471 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.498371 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-tmp-dir\") pod \"simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.498471 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.498409 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-dshm\") pod \"simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.500433 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.500416 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f59d26\"/\"kube-root-ca.crt\"" Nov 03 15:21:29.500924 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.500908 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f59d26\"/\"openshift-service-ca.crt\"" Nov 03 15:21:29.500996 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.500914 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f59d26\"/\"default-dockercfg-7hssz\"" Nov 03 15:21:29.500996 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.500992 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f59d26\"/\"simulator-w6-f59d26-kserve-self-signed-certs\"" Nov 03 15:21:29.516089 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.516060 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:29.599854 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.599814 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-dshm\") pod \"simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.600044 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.599886 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c821d166-2220-42fc-801d-62207ca877f2-tls-certs\") pod \"simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.600044 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.599937 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-model-cache\") pod \"simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.600044 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.599968 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-home\") pod \"simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.600044 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.600020 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnnzz\" (UniqueName: \"kubernetes.io/projected/c821d166-2220-42fc-801d-62207ca877f2-kube-api-access-fnnzz\") pod \"simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.600261 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.600061 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-tmp-dir\") pod \"simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.600344 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.600319 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-model-cache\") pod \"simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.600406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.600387 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-home\") pod \"simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.600517 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.600439 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-tmp-dir\") pod \"simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.602131 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.602110 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-dshm\") pod \"simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.602356 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.602339 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c821d166-2220-42fc-801d-62207ca877f2-tls-certs\") pod \"simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.670004 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.669936 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnnzz\" (UniqueName: \"kubernetes.io/projected/c821d166-2220-42fc-801d-62207ca877f2-kube-api-access-fnnzz\") pod \"simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:29.702485 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:29.702448 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:30.081425 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:30.081394 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv"] Nov 03 15:21:30.082065 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:21:30.082038 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc821d166_2220_42fc_801d_62207ca877f2.slice/crio-88df521dc8c0c973f6e7c927679b45d2cae1b2b4572fd8c8bf3bed4ff8d07362 WatchSource:0}: Error finding container 88df521dc8c0c973f6e7c927679b45d2cae1b2b4572fd8c8bf3bed4ff8d07362: Status 404 returned error can't find the container with id 88df521dc8c0c973f6e7c927679b45d2cae1b2b4572fd8c8bf3bed4ff8d07362 Nov 03 15:21:30.596008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:30.595974 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" event={"ID":"c821d166-2220-42fc-801d-62207ca877f2","Type":"ContainerStarted","Data":"fb487b339f61a65fa6470bffa59815141496ae641973c69fbbc00e01a6e5507a"} Nov 03 15:21:30.596431 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:30.596019 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" event={"ID":"c821d166-2220-42fc-801d-62207ca877f2","Type":"ContainerStarted","Data":"88df521dc8c0c973f6e7c927679b45d2cae1b2b4572fd8c8bf3bed4ff8d07362"} Nov 03 15:21:30.703180 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:30.703150 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:30.718991 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:30.718940 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podStartSLOduration=1.718927133 podStartE2EDuration="1.718927133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:21:30.717617792 +0000 UTC m=+800.560739334" watchObservedRunningTime="2026-11-03 15:21:30.718927133 +0000 UTC m=+800.562048675" Nov 03 15:21:33.348004 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:33.347967 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:33.586424 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:33.586390 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:38.202286 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:38.202246 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:39.703565 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:39.703525 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:39.716627 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:39.716597 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:40.654505 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:40.654476 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:21:41.276746 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.276706 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs"] Nov 03 15:21:41.280368 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.280345 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.312764 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.312741 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-tmp-dir\") pod \"simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.313036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.313013 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-model-cache\") pod \"simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.313203 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.313180 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-home\") pod \"simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.313252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.313232 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-tls-certs\") pod \"simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.313292 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.313261 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5jn5\" (UniqueName: \"kubernetes.io/projected/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-kube-api-access-d5jn5\") pod \"simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.313511 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.313474 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-dshm\") pod \"simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.336406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.336386 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e62ff3\"/\"simulator-w0-e62ff3-kserve-self-signed-certs\"" Nov 03 15:21:41.388370 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.388340 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e62ff3\"/\"default-dockercfg-gw6b2\"" Nov 03 15:21:41.390194 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.390174 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs"] Nov 03 15:21:41.401010 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.400979 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e62ff3\"/\"kube-root-ca.crt\"" Nov 03 15:21:41.401127 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.401014 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e62ff3\"/\"openshift-service-ca.crt\"" Nov 03 15:21:41.406819 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.406788 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:41.414209 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.414187 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-dshm\") pod \"simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.414298 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.414216 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-tmp-dir\") pod \"simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.414298 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.414287 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-model-cache\") pod \"simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.414397 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.414335 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-home\") pod \"simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.414397 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.414372 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-tls-certs\") pod \"simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.414616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.414396 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5jn5\" (UniqueName: \"kubernetes.io/projected/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-kube-api-access-d5jn5\") pod \"simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.414840 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.414818 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-tmp-dir\") pod \"simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.415167 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.415146 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-model-cache\") pod \"simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.415946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.415923 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-home\") pod \"simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.416924 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.416881 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-dshm\") pod \"simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.418015 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.417988 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-tls-certs\") pod \"simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.478113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.478085 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5jn5\" (UniqueName: \"kubernetes.io/projected/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-kube-api-access-d5jn5\") pod \"simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.593076 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.593042 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:41.980263 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:41.980236 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs"] Nov 03 15:21:41.980547 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:21:41.980526 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8216cdd_2b2d_43ee_b557_6a14c8152b3f.slice/crio-f350568b90f8d3edbd8e51e60674aedaf90a2ffecf45bf1c6ed705a8789e2dc9 WatchSource:0}: Error finding container f350568b90f8d3edbd8e51e60674aedaf90a2ffecf45bf1c6ed705a8789e2dc9: Status 404 returned error can't find the container with id f350568b90f8d3edbd8e51e60674aedaf90a2ffecf45bf1c6ed705a8789e2dc9 Nov 03 15:21:42.656714 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:42.656675 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" event={"ID":"f8216cdd-2b2d-43ee-b557-6a14c8152b3f","Type":"ContainerStarted","Data":"3cc1bfcf2badb40cae3e7708ac52d0c830e69abf2f871166f8dc4856ab85c4eb"} Nov 03 15:21:42.657109 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:42.656729 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" event={"ID":"f8216cdd-2b2d-43ee-b557-6a14c8152b3f","Type":"ContainerStarted","Data":"f350568b90f8d3edbd8e51e60674aedaf90a2ffecf45bf1c6ed705a8789e2dc9"} Nov 03 15:21:42.779534 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:42.779479 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podStartSLOduration=1.7794660869999999 podStartE2EDuration="1.779466087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:21:42.777979004 +0000 UTC m=+812.621100546" watchObservedRunningTime="2026-11-03 15:21:42.779466087 +0000 UTC m=+812.622587629" Nov 03 15:21:43.096493 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:43.096454 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:43.593370 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:43.593338 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:46.531367 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:21:46.531330 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:21:47.973985 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:47.973948 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:49.477559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:49.477520 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:51.594007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:51.593971 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:51.606351 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:51.606323 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:51.716316 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:51.716288 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:21:52.896709 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:52.896678 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:57.916920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:57.916884 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:21:59.805362 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:21:59.805329 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:22:03.023759 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:03.023698 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:22:07.918059 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:07.918027 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:22:12.837633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:12.837596 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:22:13.965303 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:13.965270 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg"] Nov 03 15:22:13.968853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:13.968832 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.027178 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.027150 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg"] Nov 03 15:22:14.077539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.077511 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-34e8c5\"/\"premium-w5-34e8c5-kserve-self-signed-certs\"" Nov 03 15:22:14.115077 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.115042 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:22:14.141361 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.141330 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-tmp-dir\") pod \"premium-w5-34e8c5-kserve-5499d7c559-v27gg\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.141520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.141370 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-home\") pod \"premium-w5-34e8c5-kserve-5499d7c559-v27gg\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.141520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.141464 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-dshm\") pod \"premium-w5-34e8c5-kserve-5499d7c559-v27gg\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.141520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.141490 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-model-cache\") pod \"premium-w5-34e8c5-kserve-5499d7c559-v27gg\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.141658 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.141525 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-tls-certs\") pod \"premium-w5-34e8c5-kserve-5499d7c559-v27gg\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.141736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.141714 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skjld\" (UniqueName: \"kubernetes.io/projected/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-kube-api-access-skjld\") pod \"premium-w5-34e8c5-kserve-5499d7c559-v27gg\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.242729 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.242656 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skjld\" (UniqueName: \"kubernetes.io/projected/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-kube-api-access-skjld\") pod \"premium-w5-34e8c5-kserve-5499d7c559-v27gg\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.242729 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.242692 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-tmp-dir\") pod \"premium-w5-34e8c5-kserve-5499d7c559-v27gg\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.242729 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.242713 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-home\") pod \"premium-w5-34e8c5-kserve-5499d7c559-v27gg\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.243021 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.242747 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-dshm\") pod \"premium-w5-34e8c5-kserve-5499d7c559-v27gg\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.243021 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.242776 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-model-cache\") pod \"premium-w5-34e8c5-kserve-5499d7c559-v27gg\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.243021 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.242819 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-tls-certs\") pod \"premium-w5-34e8c5-kserve-5499d7c559-v27gg\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.243166 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.243139 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-tmp-dir\") pod \"premium-w5-34e8c5-kserve-5499d7c559-v27gg\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.243224 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.243178 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-home\") pod \"premium-w5-34e8c5-kserve-5499d7c559-v27gg\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.243318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.243292 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-model-cache\") pod \"premium-w5-34e8c5-kserve-5499d7c559-v27gg\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.245761 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.245739 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-dshm\") pod \"premium-w5-34e8c5-kserve-5499d7c559-v27gg\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.245937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.245920 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-tls-certs\") pod \"premium-w5-34e8c5-kserve-5499d7c559-v27gg\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.307300 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.307265 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skjld\" (UniqueName: \"kubernetes.io/projected/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-kube-api-access-skjld\") pod \"premium-w5-34e8c5-kserve-5499d7c559-v27gg\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.580736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.580700 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:14.954601 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:22:14.954557 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2003121d_2d9e_41ae_a8e5_239b5af7b8d7.slice/crio-f55498cbc68b61de3d0ab36f35d40ddab5442814ff9b7965201b08edd6da4641 WatchSource:0}: Error finding container f55498cbc68b61de3d0ab36f35d40ddab5442814ff9b7965201b08edd6da4641: Status 404 returned error can't find the container with id f55498cbc68b61de3d0ab36f35d40ddab5442814ff9b7965201b08edd6da4641 Nov 03 15:22:14.955028 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:14.955007 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg"] Nov 03 15:22:15.822284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:15.822249 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" event={"ID":"2003121d-2d9e-41ae-a8e5-239b5af7b8d7","Type":"ContainerStarted","Data":"929f4994753e8be356377a8f8bf7ae1fb4d03088934901b86decbd0f0f7ba976"} Nov 03 15:22:15.822284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:15.822288 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" event={"ID":"2003121d-2d9e-41ae-a8e5-239b5af7b8d7","Type":"ContainerStarted","Data":"f55498cbc68b61de3d0ab36f35d40ddab5442814ff9b7965201b08edd6da4641"} Nov 03 15:22:15.944191 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:15.944142 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podStartSLOduration=2.944129289 podStartE2EDuration="2.944129289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:22:15.943978106 +0000 UTC m=+845.787099648" watchObservedRunningTime="2026-11-03 15:22:15.944129289 +0000 UTC m=+845.787250831" Nov 03 15:22:16.532460 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:22:16.532424 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:22:16.581259 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:16.581232 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:17.656031 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:17.656002 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:22:22.516438 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:22.516407 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:22:24.580954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:24.580863 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:24.594375 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:24.594353 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:24.875667 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:24.875567 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:22:27.413763 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:27.413718 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:22:32.310981 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:32.310949 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:22:35.688441 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:35.688403 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:22:37.233472 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:37.233436 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:22:42.135381 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:42.135345 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:22:46.529136 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:22:46.529105 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:22:47.199007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:47.198966 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:22:50.037943 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:50.037908 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:22:52.020714 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:52.020667 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:22:56.949312 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:56.949279 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:22:58.134891 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:58.134857 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8"] Nov 03 15:22:58.138259 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:58.138243 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8" Nov 03 15:22:58.169405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:58.169377 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39928067-d6jz8\" (UID: \"cc614deb-ad31-4e6b-be3f-ddea755c0c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8" Nov 03 15:22:58.169514 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:58.169415 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqbvs\" (UniqueName: \"kubernetes.io/projected/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-kube-api-access-tqbvs\") pod \"maas-api-revoke-sub-39928067-d6jz8\" (UID: \"cc614deb-ad31-4e6b-be3f-ddea755c0c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8" Nov 03 15:22:58.169514 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:58.169435 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39928067-d6jz8\" (UID: \"cc614deb-ad31-4e6b-be3f-ddea755c0c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8" Nov 03 15:22:58.196970 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:58.196948 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8"] Nov 03 15:22:58.247405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:58.247373 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:22:58.247405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:58.247397 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:22:58.270900 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:58.270872 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqbvs\" (UniqueName: \"kubernetes.io/projected/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-kube-api-access-tqbvs\") pod \"maas-api-revoke-sub-39928067-d6jz8\" (UID: \"cc614deb-ad31-4e6b-be3f-ddea755c0c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8" Nov 03 15:22:58.271032 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:58.270904 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39928067-d6jz8\" (UID: \"cc614deb-ad31-4e6b-be3f-ddea755c0c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8" Nov 03 15:22:58.271032 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:58.270988 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39928067-d6jz8\" (UID: \"cc614deb-ad31-4e6b-be3f-ddea755c0c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8" Nov 03 15:22:58.271571 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:58.271550 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39928067-d6jz8\" (UID: \"cc614deb-ad31-4e6b-be3f-ddea755c0c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8" Nov 03 15:22:58.273267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:58.273246 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39928067-d6jz8\" (UID: \"cc614deb-ad31-4e6b-be3f-ddea755c0c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8" Nov 03 15:22:58.334404 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:58.334379 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqbvs\" (UniqueName: \"kubernetes.io/projected/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-kube-api-access-tqbvs\") pod \"maas-api-revoke-sub-39928067-d6jz8\" (UID: \"cc614deb-ad31-4e6b-be3f-ddea755c0c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8" Nov 03 15:22:58.450811 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:58.450749 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8" Nov 03 15:22:58.824830 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:58.824800 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8"] Nov 03 15:22:58.825720 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:22:58.825694 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc614deb_ad31_4e6b_be3f_ddea755c0c7d.slice/crio-89f803ab92f559adcd42e27103cbcd35ea0e72a387508cdb614e4a754caff797 WatchSource:0}: Error finding container 89f803ab92f559adcd42e27103cbcd35ea0e72a387508cdb614e4a754caff797: Status 404 returned error can't find the container with id 89f803ab92f559adcd42e27103cbcd35ea0e72a387508cdb614e4a754caff797 Nov 03 15:22:59.028453 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:22:59.028423 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8" event={"ID":"cc614deb-ad31-4e6b-be3f-ddea755c0c7d","Type":"ContainerStarted","Data":"89f803ab92f559adcd42e27103cbcd35ea0e72a387508cdb614e4a754caff797"} Nov 03 15:23:00.160234 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:00.160202 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:23:01.921430 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:01.921394 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:23:04.053558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:04.053523 4557 generic.go:356] "Generic (PLEG): container finished" podID="cc614deb-ad31-4e6b-be3f-ddea755c0c7d" containerID="ec5cfaa6ed47176f7ca19ae02c33e9c41b8424e00b48fcb4e74e4b9a1b89ac60" exitCode=0 Nov 03 15:23:04.053973 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:04.053611 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8" event={"ID":"cc614deb-ad31-4e6b-be3f-ddea755c0c7d","Type":"ContainerDied","Data":"ec5cfaa6ed47176f7ca19ae02c33e9c41b8424e00b48fcb4e74e4b9a1b89ac60"} Nov 03 15:23:05.193177 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:05.193150 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8" Nov 03 15:23:05.239295 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:05.239268 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-maas-api-xks-ca\") pod \"cc614deb-ad31-4e6b-be3f-ddea755c0c7d\" (UID: \"cc614deb-ad31-4e6b-be3f-ddea755c0c7d\") " Nov 03 15:23:05.239436 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:05.239366 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-kube-api-access-tqbvs\" (UniqueName: \"kubernetes.io/projected/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-kube-api-access-tqbvs\") pod \"cc614deb-ad31-4e6b-be3f-ddea755c0c7d\" (UID: \"cc614deb-ad31-4e6b-be3f-ddea755c0c7d\") " Nov 03 15:23:05.239436 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:05.239395 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-maas-api-openshift-ca\") pod \"cc614deb-ad31-4e6b-be3f-ddea755c0c7d\" (UID: \"cc614deb-ad31-4e6b-be3f-ddea755c0c7d\") " Nov 03 15:23:05.239822 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:05.239796 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-maas-api-openshift-ca" pod "cc614deb-ad31-4e6b-be3f-ddea755c0c7d" (UID: "cc614deb-ad31-4e6b-be3f-ddea755c0c7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:23:05.241464 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:05.241444 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-maas-api-xks-ca" pod "cc614deb-ad31-4e6b-be3f-ddea755c0c7d" (UID: "cc614deb-ad31-4e6b-be3f-ddea755c0c7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:23:05.241530 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:05.241521 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-kube-api-access-tqbvs" pod "cc614deb-ad31-4e6b-be3f-ddea755c0c7d" (UID: "cc614deb-ad31-4e6b-be3f-ddea755c0c7d"). InnerVolumeSpecName "kube-api-access-tqbvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:23:05.340938 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:05.340864 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:23:05.340938 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:05.340887 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqbvs\" (UniqueName: \"kubernetes.io/projected/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-kube-api-access-tqbvs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:23:05.340938 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:05.340898 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc614deb-ad31-4e6b-be3f-ddea755c0c7d-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:23:05.432459 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:05.432422 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:23:06.063785 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:06.063757 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8" Nov 03 15:23:06.063954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:06.063784 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8" event={"ID":"cc614deb-ad31-4e6b-be3f-ddea755c0c7d","Type":"ContainerDied","Data":"89f803ab92f559adcd42e27103cbcd35ea0e72a387508cdb614e4a754caff797"} Nov 03 15:23:06.063954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:06.063834 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89f803ab92f559adcd42e27103cbcd35ea0e72a387508cdb614e4a754caff797" Nov 03 15:23:06.830138 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:06.830105 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:23:10.364290 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:10.364263 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:23:10.364726 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:10.364607 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:23:11.778936 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:11.778901 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:23:16.529888 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:23:16.529855 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:23:16.645748 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:16.645703 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:23:19.580483 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.580453 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7"] Nov 03 15:23:19.581011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.580876 4557 cpu_manager.go:414] "containerMap: removing container" podUID="cc614deb-ad31-4e6b-be3f-ddea755c0c7d" containerName="revoke-keys" Nov 03 15:23:19.581011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.580890 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc614deb-ad31-4e6b-be3f-ddea755c0c7d" containerName="revoke-keys" Nov 03 15:23:19.581011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.580967 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc614deb-ad31-4e6b-be3f-ddea755c0c7d" containerName="revoke-keys" Nov 03 15:23:19.584295 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.584278 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.679317 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.679283 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ztn6\" (UniqueName: \"kubernetes.io/projected/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-kube-api-access-2ztn6\") pod \"premium-w0-e62ff3-kserve-5bf657f878-wmtb7\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.679489 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.679397 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-tmp-dir\") pod \"premium-w0-e62ff3-kserve-5bf657f878-wmtb7\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.679489 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.679444 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-home\") pod \"premium-w0-e62ff3-kserve-5bf657f878-wmtb7\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.679622 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.679570 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-tls-certs\") pod \"premium-w0-e62ff3-kserve-5bf657f878-wmtb7\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.679685 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.679637 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-dshm\") pod \"premium-w0-e62ff3-kserve-5bf657f878-wmtb7\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.679745 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.679681 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-model-cache\") pod \"premium-w0-e62ff3-kserve-5bf657f878-wmtb7\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.692758 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.692737 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e62ff3\"/\"premium-w0-e62ff3-kserve-self-signed-certs\"" Nov 03 15:23:19.694528 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.694510 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7"] Nov 03 15:23:19.732526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.732482 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:23:19.780593 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.780542 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-tmp-dir\") pod \"premium-w0-e62ff3-kserve-5bf657f878-wmtb7\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.780756 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.780604 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-home\") pod \"premium-w0-e62ff3-kserve-5bf657f878-wmtb7\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.780756 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.780681 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-tls-certs\") pod \"premium-w0-e62ff3-kserve-5bf657f878-wmtb7\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.780756 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.780711 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-dshm\") pod \"premium-w0-e62ff3-kserve-5bf657f878-wmtb7\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.780756 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.780746 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-model-cache\") pod \"premium-w0-e62ff3-kserve-5bf657f878-wmtb7\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.781044 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.780785 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ztn6\" (UniqueName: \"kubernetes.io/projected/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-kube-api-access-2ztn6\") pod \"premium-w0-e62ff3-kserve-5bf657f878-wmtb7\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.781044 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.780971 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-tmp-dir\") pod \"premium-w0-e62ff3-kserve-5bf657f878-wmtb7\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.781147 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.781042 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-home\") pod \"premium-w0-e62ff3-kserve-5bf657f878-wmtb7\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.781205 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.781183 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-model-cache\") pod \"premium-w0-e62ff3-kserve-5bf657f878-wmtb7\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.782915 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.782886 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-dshm\") pod \"premium-w0-e62ff3-kserve-5bf657f878-wmtb7\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.783161 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.783143 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-tls-certs\") pod \"premium-w0-e62ff3-kserve-5bf657f878-wmtb7\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.844747 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.844683 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ztn6\" (UniqueName: \"kubernetes.io/projected/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-kube-api-access-2ztn6\") pod \"premium-w0-e62ff3-kserve-5bf657f878-wmtb7\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:19.896111 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:19.896083 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:20.282010 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:23:20.281964 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ea2141e_d478_4cf9_b465_cc75a7bc6dec.slice/crio-c9420314b83f9f044228ae4c5eaa8c9c39a83157b7610917c0a8b875733d4109 WatchSource:0}: Error finding container c9420314b83f9f044228ae4c5eaa8c9c39a83157b7610917c0a8b875733d4109: Status 404 returned error can't find the container with id c9420314b83f9f044228ae4c5eaa8c9c39a83157b7610917c0a8b875733d4109 Nov 03 15:23:20.284809 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:20.284785 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7"] Nov 03 15:23:21.137494 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:21.137460 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" event={"ID":"0ea2141e-d478-4cf9-b465-cc75a7bc6dec","Type":"ContainerStarted","Data":"7af0ef93537d29708a9a63466c2daacb69de36ae93fef8221e14fbe0701dfb93"} Nov 03 15:23:21.137904 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:21.137502 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" event={"ID":"0ea2141e-d478-4cf9-b465-cc75a7bc6dec","Type":"ContainerStarted","Data":"c9420314b83f9f044228ae4c5eaa8c9c39a83157b7610917c0a8b875733d4109"} Nov 03 15:23:21.260363 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:21.260307 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podStartSLOduration=2.260292875 podStartE2EDuration="2.260292875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:23:21.259725101 +0000 UTC m=+911.102846642" watchObservedRunningTime="2026-11-03 15:23:21.260292875 +0000 UTC m=+911.103414417" Nov 03 15:23:21.687860 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:21.687824 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:23:21.896996 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:21.896962 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:26.618203 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:26.618162 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:23:28.766918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:28.766878 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8"] Nov 03 15:23:28.770360 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:28.770343 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8" Nov 03 15:23:28.825182 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:28.825158 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:23:28.868506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:28.868478 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh95n\" (UniqueName: \"kubernetes.io/projected/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-kube-api-access-gh95n\") pod \"maas-api-revoke-sub-f0a24ee7-pnbp8\" (UID: \"8a8b5f8d-947f-43e9-8603-05a8b6cb4930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8" Nov 03 15:23:28.868662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:28.868525 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0a24ee7-pnbp8\" (UID: \"8a8b5f8d-947f-43e9-8603-05a8b6cb4930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8" Nov 03 15:23:28.868662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:28.868644 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0a24ee7-pnbp8\" (UID: \"8a8b5f8d-947f-43e9-8603-05a8b6cb4930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8" Nov 03 15:23:28.876717 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:28.876696 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:23:28.878565 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:28.878547 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8"] Nov 03 15:23:28.969712 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:28.969686 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0a24ee7-pnbp8\" (UID: \"8a8b5f8d-947f-43e9-8603-05a8b6cb4930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8" Nov 03 15:23:28.969839 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:28.969801 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh95n\" (UniqueName: \"kubernetes.io/projected/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-kube-api-access-gh95n\") pod \"maas-api-revoke-sub-f0a24ee7-pnbp8\" (UID: \"8a8b5f8d-947f-43e9-8603-05a8b6cb4930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8" Nov 03 15:23:28.969889 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:28.969844 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0a24ee7-pnbp8\" (UID: \"8a8b5f8d-947f-43e9-8603-05a8b6cb4930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8" Nov 03 15:23:28.970470 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:28.970448 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0a24ee7-pnbp8\" (UID: \"8a8b5f8d-947f-43e9-8603-05a8b6cb4930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8" Nov 03 15:23:28.971934 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:28.971913 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0a24ee7-pnbp8\" (UID: \"8a8b5f8d-947f-43e9-8603-05a8b6cb4930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8" Nov 03 15:23:29.032897 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:29.032846 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh95n\" (UniqueName: \"kubernetes.io/projected/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-kube-api-access-gh95n\") pod \"maas-api-revoke-sub-f0a24ee7-pnbp8\" (UID: \"8a8b5f8d-947f-43e9-8603-05a8b6cb4930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8" Nov 03 15:23:29.083598 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:29.083552 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8" Nov 03 15:23:29.455475 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:29.455446 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8"] Nov 03 15:23:29.456170 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:23:29.456146 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8b5f8d_947f_43e9_8603_05a8b6cb4930.slice/crio-f5b1943a59a1c29596acb34717ff066d2d471941c7a9ec7d062c9ca866ee3b60 WatchSource:0}: Error finding container f5b1943a59a1c29596acb34717ff066d2d471941c7a9ec7d062c9ca866ee3b60: Status 404 returned error can't find the container with id f5b1943a59a1c29596acb34717ff066d2d471941c7a9ec7d062c9ca866ee3b60 Nov 03 15:23:29.896408 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:29.896382 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:29.908576 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:29.908556 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:30.182879 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:30.182840 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8" event={"ID":"8a8b5f8d-947f-43e9-8603-05a8b6cb4930","Type":"ContainerStarted","Data":"16773dfce556eef987ec1d3b8e4d55c1ea15eeea1c3f195bb79c5af6e87293ea"} Nov 03 15:23:30.183031 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:30.182903 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8" event={"ID":"8a8b5f8d-947f-43e9-8603-05a8b6cb4930","Type":"ContainerStarted","Data":"f5b1943a59a1c29596acb34717ff066d2d471941c7a9ec7d062c9ca866ee3b60"} Nov 03 15:23:30.194217 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:30.194197 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:23:30.304793 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:30.304743 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8" podStartSLOduration=2.304730663 podStartE2EDuration="2.304730663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:23:30.303367676 +0000 UTC m=+920.146489217" watchObservedRunningTime="2026-11-03 15:23:30.304730663 +0000 UTC m=+920.147852205" Nov 03 15:23:31.188762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:31.188729 4557 generic.go:356] "Generic (PLEG): container finished" podID="8a8b5f8d-947f-43e9-8603-05a8b6cb4930" containerID="16773dfce556eef987ec1d3b8e4d55c1ea15eeea1c3f195bb79c5af6e87293ea" exitCode=0 Nov 03 15:23:31.189141 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:31.188820 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8" event={"ID":"8a8b5f8d-947f-43e9-8603-05a8b6cb4930","Type":"ContainerDied","Data":"16773dfce556eef987ec1d3b8e4d55c1ea15eeea1c3f195bb79c5af6e87293ea"} Nov 03 15:23:31.500078 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:31.500007 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:23:31.690832 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:31.690799 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:23:32.338729 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:32.338709 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8" Nov 03 15:23:32.398015 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:32.397989 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-kube-api-access-gh95n\" (UniqueName: \"kubernetes.io/projected/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-kube-api-access-gh95n\") pod \"8a8b5f8d-947f-43e9-8603-05a8b6cb4930\" (UID: \"8a8b5f8d-947f-43e9-8603-05a8b6cb4930\") " Nov 03 15:23:32.398131 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:32.398036 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-maas-api-openshift-ca\") pod \"8a8b5f8d-947f-43e9-8603-05a8b6cb4930\" (UID: \"8a8b5f8d-947f-43e9-8603-05a8b6cb4930\") " Nov 03 15:23:32.398131 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:32.398082 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-maas-api-xks-ca\") pod \"8a8b5f8d-947f-43e9-8603-05a8b6cb4930\" (UID: \"8a8b5f8d-947f-43e9-8603-05a8b6cb4930\") " Nov 03 15:23:32.398396 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:32.398369 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-maas-api-openshift-ca" pod "8a8b5f8d-947f-43e9-8603-05a8b6cb4930" (UID: "8a8b5f8d-947f-43e9-8603-05a8b6cb4930"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:23:32.400014 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:32.399991 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-maas-api-xks-ca" pod "8a8b5f8d-947f-43e9-8603-05a8b6cb4930" (UID: "8a8b5f8d-947f-43e9-8603-05a8b6cb4930"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:23:32.400106 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:32.400081 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-kube-api-access-gh95n" pod "8a8b5f8d-947f-43e9-8603-05a8b6cb4930" (UID: "8a8b5f8d-947f-43e9-8603-05a8b6cb4930"). InnerVolumeSpecName "kube-api-access-gh95n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:23:32.499560 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:32.499493 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh95n\" (UniqueName: \"kubernetes.io/projected/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-kube-api-access-gh95n\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:23:32.499560 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:32.499519 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:23:32.499560 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:32.499530 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a8b5f8d-947f-43e9-8603-05a8b6cb4930-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:23:33.199860 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:33.199829 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8" Nov 03 15:23:33.199860 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:33.199843 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8" event={"ID":"8a8b5f8d-947f-43e9-8603-05a8b6cb4930","Type":"ContainerDied","Data":"f5b1943a59a1c29596acb34717ff066d2d471941c7a9ec7d062c9ca866ee3b60"} Nov 03 15:23:33.200109 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:33.199919 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5b1943a59a1c29596acb34717ff066d2d471941c7a9ec7d062c9ca866ee3b60" Nov 03 15:23:34.247276 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.247248 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4"] Nov 03 15:23:34.247806 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.247788 4557 cpu_manager.go:414] "containerMap: removing container" podUID="8a8b5f8d-947f-43e9-8603-05a8b6cb4930" containerName="revoke-keys" Nov 03 15:23:34.247908 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.247809 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a8b5f8d-947f-43e9-8603-05a8b6cb4930" containerName="revoke-keys" Nov 03 15:23:34.247998 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.247985 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a8b5f8d-947f-43e9-8603-05a8b6cb4930" containerName="revoke-keys" Nov 03 15:23:34.256932 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.256911 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4" Nov 03 15:23:34.310230 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.310200 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4"] Nov 03 15:23:34.317149 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.317123 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tclqg\" (UniqueName: \"kubernetes.io/projected/47a98824-02da-44d2-95d6-971e5fb4f3a2-kube-api-access-tclqg\") pod \"maas-api-revoke-sub-5b726b97-8pmb4\" (UID: \"47a98824-02da-44d2-95d6-971e5fb4f3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4" Nov 03 15:23:34.317273 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.317170 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47a98824-02da-44d2-95d6-971e5fb4f3a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b726b97-8pmb4\" (UID: \"47a98824-02da-44d2-95d6-971e5fb4f3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4" Nov 03 15:23:34.317326 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.317306 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47a98824-02da-44d2-95d6-971e5fb4f3a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b726b97-8pmb4\" (UID: \"47a98824-02da-44d2-95d6-971e5fb4f3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4" Nov 03 15:23:34.356803 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.356776 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:23:34.357102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.357062 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:23:34.418550 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.418484 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47a98824-02da-44d2-95d6-971e5fb4f3a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b726b97-8pmb4\" (UID: \"47a98824-02da-44d2-95d6-971e5fb4f3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4" Nov 03 15:23:34.418799 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.418627 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tclqg\" (UniqueName: \"kubernetes.io/projected/47a98824-02da-44d2-95d6-971e5fb4f3a2-kube-api-access-tclqg\") pod \"maas-api-revoke-sub-5b726b97-8pmb4\" (UID: \"47a98824-02da-44d2-95d6-971e5fb4f3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4" Nov 03 15:23:34.418799 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.418671 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47a98824-02da-44d2-95d6-971e5fb4f3a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b726b97-8pmb4\" (UID: \"47a98824-02da-44d2-95d6-971e5fb4f3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4" Nov 03 15:23:34.419537 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.419504 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47a98824-02da-44d2-95d6-971e5fb4f3a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b726b97-8pmb4\" (UID: \"47a98824-02da-44d2-95d6-971e5fb4f3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4" Nov 03 15:23:34.421558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.421537 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47a98824-02da-44d2-95d6-971e5fb4f3a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b726b97-8pmb4\" (UID: \"47a98824-02da-44d2-95d6-971e5fb4f3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4" Nov 03 15:23:34.481597 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.481551 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tclqg\" (UniqueName: \"kubernetes.io/projected/47a98824-02da-44d2-95d6-971e5fb4f3a2-kube-api-access-tclqg\") pod \"maas-api-revoke-sub-5b726b97-8pmb4\" (UID: \"47a98824-02da-44d2-95d6-971e5fb4f3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4" Nov 03 15:23:34.569403 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.569374 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4" Nov 03 15:23:34.950620 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:34.950571 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4"] Nov 03 15:23:34.952917 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:23:34.952893 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a98824_02da_44d2_95d6_971e5fb4f3a2.slice/crio-eb4d7122ca59d4adb16d77d6d69fd06d8963cdabee6a5cab4e34798c610456c8 WatchSource:0}: Error finding container eb4d7122ca59d4adb16d77d6d69fd06d8963cdabee6a5cab4e34798c610456c8: Status 404 returned error can't find the container with id eb4d7122ca59d4adb16d77d6d69fd06d8963cdabee6a5cab4e34798c610456c8 Nov 03 15:23:35.216066 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:35.215966 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4" event={"ID":"47a98824-02da-44d2-95d6-971e5fb4f3a2","Type":"ContainerStarted","Data":"eb4d7122ca59d4adb16d77d6d69fd06d8963cdabee6a5cab4e34798c610456c8"} Nov 03 15:23:36.222690 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:36.222577 4557 generic.go:356] "Generic (PLEG): container finished" podID="47a98824-02da-44d2-95d6-971e5fb4f3a2" containerID="126999c00321b0fe194e3ce9f3f51cc133eea72cefd9cb73aa1cea9d9818f95e" exitCode=0 Nov 03 15:23:36.223036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:36.222684 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4" event={"ID":"47a98824-02da-44d2-95d6-971e5fb4f3a2","Type":"ContainerDied","Data":"126999c00321b0fe194e3ce9f3f51cc133eea72cefd9cb73aa1cea9d9818f95e"} Nov 03 15:23:36.423276 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:36.423240 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:23:36.677469 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:36.677421 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:23:36.872942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:36.872907 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:23:37.364904 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:37.364881 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4" Nov 03 15:23:37.447861 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:37.447828 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a98824-02da-44d2-95d6-971e5fb4f3a2-kube-api-access-tclqg\" (UniqueName: \"kubernetes.io/projected/47a98824-02da-44d2-95d6-971e5fb4f3a2-kube-api-access-tclqg\") pod \"47a98824-02da-44d2-95d6-971e5fb4f3a2\" (UID: \"47a98824-02da-44d2-95d6-971e5fb4f3a2\") " Nov 03 15:23:37.448022 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:37.447871 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47a98824-02da-44d2-95d6-971e5fb4f3a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47a98824-02da-44d2-95d6-971e5fb4f3a2-maas-api-openshift-ca\") pod \"47a98824-02da-44d2-95d6-971e5fb4f3a2\" (UID: \"47a98824-02da-44d2-95d6-971e5fb4f3a2\") " Nov 03 15:23:37.448022 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:37.447952 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47a98824-02da-44d2-95d6-971e5fb4f3a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47a98824-02da-44d2-95d6-971e5fb4f3a2-maas-api-xks-ca\") pod \"47a98824-02da-44d2-95d6-971e5fb4f3a2\" (UID: \"47a98824-02da-44d2-95d6-971e5fb4f3a2\") " Nov 03 15:23:37.448290 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:37.448263 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a98824-02da-44d2-95d6-971e5fb4f3a2-maas-api-openshift-ca" pod "47a98824-02da-44d2-95d6-971e5fb4f3a2" (UID: "47a98824-02da-44d2-95d6-971e5fb4f3a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:23:37.449867 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:37.449842 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47a98824-02da-44d2-95d6-971e5fb4f3a2-maas-api-xks-ca" pod "47a98824-02da-44d2-95d6-971e5fb4f3a2" (UID: "47a98824-02da-44d2-95d6-971e5fb4f3a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:23:37.449970 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:37.449910 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a98824-02da-44d2-95d6-971e5fb4f3a2-kube-api-access-tclqg" pod "47a98824-02da-44d2-95d6-971e5fb4f3a2" (UID: "47a98824-02da-44d2-95d6-971e5fb4f3a2"). InnerVolumeSpecName "kube-api-access-tclqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:23:37.549107 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:37.549064 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tclqg\" (UniqueName: \"kubernetes.io/projected/47a98824-02da-44d2-95d6-971e5fb4f3a2-kube-api-access-tclqg\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:23:37.549107 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:37.549105 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47a98824-02da-44d2-95d6-971e5fb4f3a2-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:23:37.549285 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:37.549119 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47a98824-02da-44d2-95d6-971e5fb4f3a2-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:23:38.233254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:38.233225 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4" Nov 03 15:23:38.233254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:38.233234 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4" event={"ID":"47a98824-02da-44d2-95d6-971e5fb4f3a2","Type":"ContainerDied","Data":"eb4d7122ca59d4adb16d77d6d69fd06d8963cdabee6a5cab4e34798c610456c8"} Nov 03 15:23:38.233474 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:38.233275 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb4d7122ca59d4adb16d77d6d69fd06d8963cdabee6a5cab4e34798c610456c8" Nov 03 15:23:41.230149 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:41.230108 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:23:44.411842 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:44.411809 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl"] Nov 03 15:23:44.412205 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:44.412191 4557 cpu_manager.go:414] "containerMap: removing container" podUID="47a98824-02da-44d2-95d6-971e5fb4f3a2" containerName="revoke-keys" Nov 03 15:23:44.412205 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:44.412202 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a98824-02da-44d2-95d6-971e5fb4f3a2" containerName="revoke-keys" Nov 03 15:23:44.412293 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:44.412261 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a98824-02da-44d2-95d6-971e5fb4f3a2" containerName="revoke-keys" Nov 03 15:23:44.415572 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:44.415552 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl" Nov 03 15:23:44.470512 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:44.470486 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:23:44.473155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:44.473129 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl"] Nov 03 15:23:44.514569 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:44.514539 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/969b81a7-ad40-477e-9032-39e6532b16fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-617ae85d-x7trl\" (UID: \"969b81a7-ad40-477e-9032-39e6532b16fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl" Nov 03 15:23:44.514706 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:44.514656 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/969b81a7-ad40-477e-9032-39e6532b16fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-617ae85d-x7trl\" (UID: \"969b81a7-ad40-477e-9032-39e6532b16fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl" Nov 03 15:23:44.514747 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:44.514725 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czl8l\" (UniqueName: \"kubernetes.io/projected/969b81a7-ad40-477e-9032-39e6532b16fd-kube-api-access-czl8l\") pod \"maas-api-revoke-sub-617ae85d-x7trl\" (UID: \"969b81a7-ad40-477e-9032-39e6532b16fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl" Nov 03 15:23:44.520823 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:44.520805 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:23:44.615362 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:44.615334 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/969b81a7-ad40-477e-9032-39e6532b16fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-617ae85d-x7trl\" (UID: \"969b81a7-ad40-477e-9032-39e6532b16fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl" Nov 03 15:23:44.615511 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:44.615381 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/969b81a7-ad40-477e-9032-39e6532b16fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-617ae85d-x7trl\" (UID: \"969b81a7-ad40-477e-9032-39e6532b16fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl" Nov 03 15:23:44.615602 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:44.615560 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czl8l\" (UniqueName: \"kubernetes.io/projected/969b81a7-ad40-477e-9032-39e6532b16fd-kube-api-access-czl8l\") pod \"maas-api-revoke-sub-617ae85d-x7trl\" (UID: \"969b81a7-ad40-477e-9032-39e6532b16fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl" Nov 03 15:23:44.616099 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:44.616073 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/969b81a7-ad40-477e-9032-39e6532b16fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-617ae85d-x7trl\" (UID: \"969b81a7-ad40-477e-9032-39e6532b16fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl" Nov 03 15:23:44.617652 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:44.617634 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/969b81a7-ad40-477e-9032-39e6532b16fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-617ae85d-x7trl\" (UID: \"969b81a7-ad40-477e-9032-39e6532b16fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl" Nov 03 15:23:44.677394 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:44.677334 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czl8l\" (UniqueName: \"kubernetes.io/projected/969b81a7-ad40-477e-9032-39e6532b16fd-kube-api-access-czl8l\") pod \"maas-api-revoke-sub-617ae85d-x7trl\" (UID: \"969b81a7-ad40-477e-9032-39e6532b16fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl" Nov 03 15:23:44.727966 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:44.727944 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl" Nov 03 15:23:45.109938 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:45.109903 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl"] Nov 03 15:23:45.116504 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:23:45.116474 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969b81a7_ad40_477e_9032_39e6532b16fd.slice/crio-b55e76db721b7248f24ec31b74684d0ee602ca87e04d0d5a887a19e8a25c0ec5 WatchSource:0}: Error finding container b55e76db721b7248f24ec31b74684d0ee602ca87e04d0d5a887a19e8a25c0ec5: Status 404 returned error can't find the container with id b55e76db721b7248f24ec31b74684d0ee602ca87e04d0d5a887a19e8a25c0ec5 Nov 03 15:23:45.267183 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:45.267150 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl" event={"ID":"969b81a7-ad40-477e-9032-39e6532b16fd","Type":"ContainerStarted","Data":"b55e76db721b7248f24ec31b74684d0ee602ca87e04d0d5a887a19e8a25c0ec5"} Nov 03 15:23:46.113370 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:46.113336 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:23:46.273169 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:46.273086 4557 generic.go:356] "Generic (PLEG): container finished" podID="969b81a7-ad40-477e-9032-39e6532b16fd" containerID="53d902445fd73a8acaacdfde00d625a0c44e20e35b7cfbcfec27bb1c6ce4f1a5" exitCode=0 Nov 03 15:23:46.273169 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:46.273163 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl" event={"ID":"969b81a7-ad40-477e-9032-39e6532b16fd","Type":"ContainerDied","Data":"53d902445fd73a8acaacdfde00d625a0c44e20e35b7cfbcfec27bb1c6ce4f1a5"} Nov 03 15:23:46.530631 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:23:46.530530 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:23:47.415190 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:47.415167 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl" Nov 03 15:23:47.543571 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:47.543486 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/969b81a7-ad40-477e-9032-39e6532b16fd-kube-api-access-czl8l\" (UniqueName: \"kubernetes.io/projected/969b81a7-ad40-477e-9032-39e6532b16fd-kube-api-access-czl8l\") pod \"969b81a7-ad40-477e-9032-39e6532b16fd\" (UID: \"969b81a7-ad40-477e-9032-39e6532b16fd\") " Nov 03 15:23:47.543742 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:47.543627 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/969b81a7-ad40-477e-9032-39e6532b16fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/969b81a7-ad40-477e-9032-39e6532b16fd-maas-api-openshift-ca\") pod \"969b81a7-ad40-477e-9032-39e6532b16fd\" (UID: \"969b81a7-ad40-477e-9032-39e6532b16fd\") " Nov 03 15:23:47.543742 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:47.543670 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/969b81a7-ad40-477e-9032-39e6532b16fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/969b81a7-ad40-477e-9032-39e6532b16fd-maas-api-xks-ca\") pod \"969b81a7-ad40-477e-9032-39e6532b16fd\" (UID: \"969b81a7-ad40-477e-9032-39e6532b16fd\") " Nov 03 15:23:47.544199 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:47.544171 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/969b81a7-ad40-477e-9032-39e6532b16fd-maas-api-openshift-ca" pod "969b81a7-ad40-477e-9032-39e6532b16fd" (UID: "969b81a7-ad40-477e-9032-39e6532b16fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:23:47.545926 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:47.545895 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/969b81a7-ad40-477e-9032-39e6532b16fd-maas-api-xks-ca" pod "969b81a7-ad40-477e-9032-39e6532b16fd" (UID: "969b81a7-ad40-477e-9032-39e6532b16fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:23:47.546177 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:47.546155 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969b81a7-ad40-477e-9032-39e6532b16fd-kube-api-access-czl8l" pod "969b81a7-ad40-477e-9032-39e6532b16fd" (UID: "969b81a7-ad40-477e-9032-39e6532b16fd"). InnerVolumeSpecName "kube-api-access-czl8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:23:47.644942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:47.644913 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czl8l\" (UniqueName: \"kubernetes.io/projected/969b81a7-ad40-477e-9032-39e6532b16fd-kube-api-access-czl8l\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:23:47.644942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:47.644940 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/969b81a7-ad40-477e-9032-39e6532b16fd-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:23:47.644942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:47.644951 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/969b81a7-ad40-477e-9032-39e6532b16fd-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:23:48.284409 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:48.284382 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl" Nov 03 15:23:48.284783 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:48.284754 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl" event={"ID":"969b81a7-ad40-477e-9032-39e6532b16fd","Type":"ContainerDied","Data":"b55e76db721b7248f24ec31b74684d0ee602ca87e04d0d5a887a19e8a25c0ec5"} Nov 03 15:23:48.284892 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:48.284801 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b55e76db721b7248f24ec31b74684d0ee602ca87e04d0d5a887a19e8a25c0ec5" Nov 03 15:23:50.010649 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.010613 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz"] Nov 03 15:23:50.011062 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.011048 4557 cpu_manager.go:414] "containerMap: removing container" podUID="969b81a7-ad40-477e-9032-39e6532b16fd" containerName="revoke-keys" Nov 03 15:23:50.011106 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.011063 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969b81a7-ad40-477e-9032-39e6532b16fd" containerName="revoke-keys" Nov 03 15:23:50.011150 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.011140 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="969b81a7-ad40-477e-9032-39e6532b16fd" containerName="revoke-keys" Nov 03 15:23:50.014329 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.014308 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz" Nov 03 15:23:50.069034 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.069006 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7zkb\" (UniqueName: \"kubernetes.io/projected/5864b770-7acf-4498-a7fd-2319cabe6307-kube-api-access-t7zkb\") pod \"maas-api-revoke-sub-fb88ad8c-wxfgz\" (UID: \"5864b770-7acf-4498-a7fd-2319cabe6307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz" Nov 03 15:23:50.069201 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.069114 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5864b770-7acf-4498-a7fd-2319cabe6307-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb88ad8c-wxfgz\" (UID: \"5864b770-7acf-4498-a7fd-2319cabe6307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz" Nov 03 15:23:50.069257 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.069195 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5864b770-7acf-4498-a7fd-2319cabe6307-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb88ad8c-wxfgz\" (UID: \"5864b770-7acf-4498-a7fd-2319cabe6307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz" Nov 03 15:23:50.072254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.072230 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz"] Nov 03 15:23:50.119280 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.119257 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:23:50.119440 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.119336 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:23:50.169994 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.169964 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7zkb\" (UniqueName: \"kubernetes.io/projected/5864b770-7acf-4498-a7fd-2319cabe6307-kube-api-access-t7zkb\") pod \"maas-api-revoke-sub-fb88ad8c-wxfgz\" (UID: \"5864b770-7acf-4498-a7fd-2319cabe6307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz" Nov 03 15:23:50.170151 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.170032 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5864b770-7acf-4498-a7fd-2319cabe6307-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb88ad8c-wxfgz\" (UID: \"5864b770-7acf-4498-a7fd-2319cabe6307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz" Nov 03 15:23:50.170151 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.170094 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5864b770-7acf-4498-a7fd-2319cabe6307-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb88ad8c-wxfgz\" (UID: \"5864b770-7acf-4498-a7fd-2319cabe6307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz" Nov 03 15:23:50.170748 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.170717 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5864b770-7acf-4498-a7fd-2319cabe6307-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb88ad8c-wxfgz\" (UID: \"5864b770-7acf-4498-a7fd-2319cabe6307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz" Nov 03 15:23:50.172335 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.172318 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5864b770-7acf-4498-a7fd-2319cabe6307-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb88ad8c-wxfgz\" (UID: \"5864b770-7acf-4498-a7fd-2319cabe6307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz" Nov 03 15:23:50.233939 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.233916 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7zkb\" (UniqueName: \"kubernetes.io/projected/5864b770-7acf-4498-a7fd-2319cabe6307-kube-api-access-t7zkb\") pod \"maas-api-revoke-sub-fb88ad8c-wxfgz\" (UID: \"5864b770-7acf-4498-a7fd-2319cabe6307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz" Nov 03 15:23:50.326603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.326556 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz" Nov 03 15:23:50.700634 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:50.700609 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz"] Nov 03 15:23:50.702601 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:23:50.702549 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5864b770_7acf_4498_a7fd_2319cabe6307.slice/crio-b58973a3dff6632d1de88631b34c827cedf15e8f44ea266438e2d9b73ea13801 WatchSource:0}: Error finding container b58973a3dff6632d1de88631b34c827cedf15e8f44ea266438e2d9b73ea13801: Status 404 returned error can't find the container with id b58973a3dff6632d1de88631b34c827cedf15e8f44ea266438e2d9b73ea13801 Nov 03 15:23:51.027382 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:51.027355 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:23:51.299685 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:51.299652 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz" event={"ID":"5864b770-7acf-4498-a7fd-2319cabe6307","Type":"ContainerStarted","Data":"5d7b363704716993df9ef96f05febb67c5657ec2dbec900a9e741184465353c7"} Nov 03 15:23:51.299860 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:51.299700 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz" event={"ID":"5864b770-7acf-4498-a7fd-2319cabe6307","Type":"ContainerStarted","Data":"b58973a3dff6632d1de88631b34c827cedf15e8f44ea266438e2d9b73ea13801"} Nov 03 15:23:51.417177 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:51.417121 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz" podStartSLOduration=2.417107293 podStartE2EDuration="2.417107293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:23:51.415931482 +0000 UTC m=+941.259053021" watchObservedRunningTime="2026-11-03 15:23:51.417107293 +0000 UTC m=+941.260228834" Nov 03 15:23:52.305885 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:52.305849 4557 generic.go:356] "Generic (PLEG): container finished" podID="5864b770-7acf-4498-a7fd-2319cabe6307" containerID="5d7b363704716993df9ef96f05febb67c5657ec2dbec900a9e741184465353c7" exitCode=0 Nov 03 15:23:52.306283 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:52.305926 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz" event={"ID":"5864b770-7acf-4498-a7fd-2319cabe6307","Type":"ContainerDied","Data":"5d7b363704716993df9ef96f05febb67c5657ec2dbec900a9e741184465353c7"} Nov 03 15:23:53.445435 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:53.445411 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz" Nov 03 15:23:53.504815 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:53.504783 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5864b770-7acf-4498-a7fd-2319cabe6307-kube-api-access-t7zkb\" (UniqueName: \"kubernetes.io/projected/5864b770-7acf-4498-a7fd-2319cabe6307-kube-api-access-t7zkb\") pod \"5864b770-7acf-4498-a7fd-2319cabe6307\" (UID: \"5864b770-7acf-4498-a7fd-2319cabe6307\") " Nov 03 15:23:53.504988 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:53.504955 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5864b770-7acf-4498-a7fd-2319cabe6307-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5864b770-7acf-4498-a7fd-2319cabe6307-maas-api-xks-ca\") pod \"5864b770-7acf-4498-a7fd-2319cabe6307\" (UID: \"5864b770-7acf-4498-a7fd-2319cabe6307\") " Nov 03 15:23:53.505027 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:53.504985 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5864b770-7acf-4498-a7fd-2319cabe6307-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5864b770-7acf-4498-a7fd-2319cabe6307-maas-api-openshift-ca\") pod \"5864b770-7acf-4498-a7fd-2319cabe6307\" (UID: \"5864b770-7acf-4498-a7fd-2319cabe6307\") " Nov 03 15:23:53.505395 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:53.505367 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5864b770-7acf-4498-a7fd-2319cabe6307-maas-api-openshift-ca" pod "5864b770-7acf-4498-a7fd-2319cabe6307" (UID: "5864b770-7acf-4498-a7fd-2319cabe6307"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:23:53.506838 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:53.506813 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5864b770-7acf-4498-a7fd-2319cabe6307-kube-api-access-t7zkb" pod "5864b770-7acf-4498-a7fd-2319cabe6307" (UID: "5864b770-7acf-4498-a7fd-2319cabe6307"). InnerVolumeSpecName "kube-api-access-t7zkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:23:53.507020 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:53.506999 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5864b770-7acf-4498-a7fd-2319cabe6307-maas-api-xks-ca" pod "5864b770-7acf-4498-a7fd-2319cabe6307" (UID: "5864b770-7acf-4498-a7fd-2319cabe6307"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:23:53.606139 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:53.606064 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5864b770-7acf-4498-a7fd-2319cabe6307-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:23:53.606139 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:53.606091 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5864b770-7acf-4498-a7fd-2319cabe6307-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:23:53.606139 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:53.606100 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7zkb\" (UniqueName: \"kubernetes.io/projected/5864b770-7acf-4498-a7fd-2319cabe6307-kube-api-access-t7zkb\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:23:54.316641 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:54.316609 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz" Nov 03 15:23:54.316641 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:54.316630 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz" event={"ID":"5864b770-7acf-4498-a7fd-2319cabe6307","Type":"ContainerDied","Data":"b58973a3dff6632d1de88631b34c827cedf15e8f44ea266438e2d9b73ea13801"} Nov 03 15:23:54.316813 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:54.316678 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b58973a3dff6632d1de88631b34c827cedf15e8f44ea266438e2d9b73ea13801" Nov 03 15:23:55.697473 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:55.697431 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:23:56.213441 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:23:56.213405 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:00.509609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:00.509549 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:01.142170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:01.142131 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:05.979568 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:05.979532 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d"] Nov 03 15:24:05.980491 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:05.980464 4557 cpu_manager.go:414] "containerMap: removing container" podUID="5864b770-7acf-4498-a7fd-2319cabe6307" containerName="revoke-keys" Nov 03 15:24:05.980491 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:05.980492 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5864b770-7acf-4498-a7fd-2319cabe6307" containerName="revoke-keys" Nov 03 15:24:05.980855 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:05.980840 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="5864b770-7acf-4498-a7fd-2319cabe6307" containerName="revoke-keys" Nov 03 15:24:05.985929 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:05.985901 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d" Nov 03 15:24:06.042021 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:06.041985 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d"] Nov 03 15:24:06.046560 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:06.046527 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8eeed72-3d78-49e0-9e5d-9a62420298b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-823513b6-5nl9d\" (UID: \"d8eeed72-3d78-49e0-9e5d-9a62420298b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d" Nov 03 15:24:06.046748 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:06.046723 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8eeed72-3d78-49e0-9e5d-9a62420298b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-823513b6-5nl9d\" (UID: \"d8eeed72-3d78-49e0-9e5d-9a62420298b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d" Nov 03 15:24:06.046826 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:06.046764 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pdxm\" (UniqueName: \"kubernetes.io/projected/d8eeed72-3d78-49e0-9e5d-9a62420298b9-kube-api-access-9pdxm\") pod \"maas-api-revoke-sub-823513b6-5nl9d\" (UID: \"d8eeed72-3d78-49e0-9e5d-9a62420298b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d" Nov 03 15:24:06.065548 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:06.065523 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:24:06.088785 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:06.088762 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:24:06.127630 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:06.127571 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:06.148017 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:06.147985 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8eeed72-3d78-49e0-9e5d-9a62420298b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-823513b6-5nl9d\" (UID: \"d8eeed72-3d78-49e0-9e5d-9a62420298b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d" Nov 03 15:24:06.148017 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:06.148015 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdxm\" (UniqueName: \"kubernetes.io/projected/d8eeed72-3d78-49e0-9e5d-9a62420298b9-kube-api-access-9pdxm\") pod \"maas-api-revoke-sub-823513b6-5nl9d\" (UID: \"d8eeed72-3d78-49e0-9e5d-9a62420298b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d" Nov 03 15:24:06.148215 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:06.148091 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8eeed72-3d78-49e0-9e5d-9a62420298b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-823513b6-5nl9d\" (UID: \"d8eeed72-3d78-49e0-9e5d-9a62420298b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d" Nov 03 15:24:06.148801 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:06.148780 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8eeed72-3d78-49e0-9e5d-9a62420298b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-823513b6-5nl9d\" (UID: \"d8eeed72-3d78-49e0-9e5d-9a62420298b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d" Nov 03 15:24:06.150340 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:06.150314 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8eeed72-3d78-49e0-9e5d-9a62420298b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-823513b6-5nl9d\" (UID: \"d8eeed72-3d78-49e0-9e5d-9a62420298b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d" Nov 03 15:24:06.210014 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:06.209987 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pdxm\" (UniqueName: \"kubernetes.io/projected/d8eeed72-3d78-49e0-9e5d-9a62420298b9-kube-api-access-9pdxm\") pod \"maas-api-revoke-sub-823513b6-5nl9d\" (UID: \"d8eeed72-3d78-49e0-9e5d-9a62420298b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d" Nov 03 15:24:06.298552 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:06.298526 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d" Nov 03 15:24:06.670045 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:06.670019 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d"] Nov 03 15:24:06.671489 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:24:06.671462 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8eeed72_3d78_49e0_9e5d_9a62420298b9.slice/crio-2617809217f02efefe2c2dc93a489ff54bde424c5b5e8a43b1a052e8e1090715 WatchSource:0}: Error finding container 2617809217f02efefe2c2dc93a489ff54bde424c5b5e8a43b1a052e8e1090715: Status 404 returned error can't find the container with id 2617809217f02efefe2c2dc93a489ff54bde424c5b5e8a43b1a052e8e1090715 Nov 03 15:24:07.384139 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:07.384095 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d" event={"ID":"d8eeed72-3d78-49e0-9e5d-9a62420298b9","Type":"ContainerStarted","Data":"0b68ee73cb1aa17896dffc150ebf565fa45f5f6988c126f89f9f7130b5de3250"} Nov 03 15:24:07.384512 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:07.384152 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d" event={"ID":"d8eeed72-3d78-49e0-9e5d-9a62420298b9","Type":"ContainerStarted","Data":"2617809217f02efefe2c2dc93a489ff54bde424c5b5e8a43b1a052e8e1090715"} Nov 03 15:24:07.502260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:07.502210 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d" podStartSLOduration=2.502197125 podStartE2EDuration="2.502197125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:24:07.501092392 +0000 UTC m=+957.344213943" watchObservedRunningTime="2026-11-03 15:24:07.502197125 +0000 UTC m=+957.345318666" Nov 03 15:24:08.390257 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:08.390222 4557 generic.go:356] "Generic (PLEG): container finished" podID="d8eeed72-3d78-49e0-9e5d-9a62420298b9" containerID="0b68ee73cb1aa17896dffc150ebf565fa45f5f6988c126f89f9f7130b5de3250" exitCode=0 Nov 03 15:24:08.390669 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:08.390306 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d" event={"ID":"d8eeed72-3d78-49e0-9e5d-9a62420298b9","Type":"ContainerDied","Data":"0b68ee73cb1aa17896dffc150ebf565fa45f5f6988c126f89f9f7130b5de3250"} Nov 03 15:24:09.542511 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:09.542486 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d" Nov 03 15:24:09.580892 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:09.580867 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8eeed72-3d78-49e0-9e5d-9a62420298b9-kube-api-access-9pdxm\" (UniqueName: \"kubernetes.io/projected/d8eeed72-3d78-49e0-9e5d-9a62420298b9-kube-api-access-9pdxm\") pod \"d8eeed72-3d78-49e0-9e5d-9a62420298b9\" (UID: \"d8eeed72-3d78-49e0-9e5d-9a62420298b9\") " Nov 03 15:24:09.581016 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:09.580934 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8eeed72-3d78-49e0-9e5d-9a62420298b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8eeed72-3d78-49e0-9e5d-9a62420298b9-maas-api-xks-ca\") pod \"d8eeed72-3d78-49e0-9e5d-9a62420298b9\" (UID: \"d8eeed72-3d78-49e0-9e5d-9a62420298b9\") " Nov 03 15:24:09.581016 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:09.580975 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8eeed72-3d78-49e0-9e5d-9a62420298b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8eeed72-3d78-49e0-9e5d-9a62420298b9-maas-api-openshift-ca\") pod \"d8eeed72-3d78-49e0-9e5d-9a62420298b9\" (UID: \"d8eeed72-3d78-49e0-9e5d-9a62420298b9\") " Nov 03 15:24:09.581346 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:09.581320 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8eeed72-3d78-49e0-9e5d-9a62420298b9-maas-api-openshift-ca" pod "d8eeed72-3d78-49e0-9e5d-9a62420298b9" (UID: "d8eeed72-3d78-49e0-9e5d-9a62420298b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:24:09.582859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:09.582829 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8eeed72-3d78-49e0-9e5d-9a62420298b9-kube-api-access-9pdxm" pod "d8eeed72-3d78-49e0-9e5d-9a62420298b9" (UID: "d8eeed72-3d78-49e0-9e5d-9a62420298b9"). InnerVolumeSpecName "kube-api-access-9pdxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:24:09.582945 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:09.582906 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8eeed72-3d78-49e0-9e5d-9a62420298b9-maas-api-xks-ca" pod "d8eeed72-3d78-49e0-9e5d-9a62420298b9" (UID: "d8eeed72-3d78-49e0-9e5d-9a62420298b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:24:09.682560 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:09.682487 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8eeed72-3d78-49e0-9e5d-9a62420298b9-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:09.682560 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:09.682522 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8eeed72-3d78-49e0-9e5d-9a62420298b9-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:09.682560 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:09.682541 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pdxm\" (UniqueName: \"kubernetes.io/projected/d8eeed72-3d78-49e0-9e5d-9a62420298b9-kube-api-access-9pdxm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:10.401358 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:10.401327 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d" event={"ID":"d8eeed72-3d78-49e0-9e5d-9a62420298b9","Type":"ContainerDied","Data":"2617809217f02efefe2c2dc93a489ff54bde424c5b5e8a43b1a052e8e1090715"} Nov 03 15:24:10.401520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:10.401397 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2617809217f02efefe2c2dc93a489ff54bde424c5b5e8a43b1a052e8e1090715" Nov 03 15:24:10.401520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:10.401337 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d" Nov 03 15:24:10.942880 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:10.942837 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:15.810174 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:15.810137 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:16.530042 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:24:16.530004 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:24:16.800093 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:16.800058 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj"] Nov 03 15:24:16.800498 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:16.800485 4557 cpu_manager.go:414] "containerMap: removing container" podUID="d8eeed72-3d78-49e0-9e5d-9a62420298b9" containerName="revoke-keys" Nov 03 15:24:16.800552 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:16.800499 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8eeed72-3d78-49e0-9e5d-9a62420298b9" containerName="revoke-keys" Nov 03 15:24:16.800624 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:16.800565 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8eeed72-3d78-49e0-9e5d-9a62420298b9" containerName="revoke-keys" Nov 03 15:24:16.803809 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:16.803792 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj" Nov 03 15:24:16.849776 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:16.849737 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f383e1be-904d-40d8-8f33-df320987b7cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76f64878-psglj\" (UID: \"f383e1be-904d-40d8-8f33-df320987b7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj" Nov 03 15:24:16.850191 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:16.849810 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vn9g\" (UniqueName: \"kubernetes.io/projected/f383e1be-904d-40d8-8f33-df320987b7cb-kube-api-access-9vn9g\") pod \"maas-api-revoke-sub-76f64878-psglj\" (UID: \"f383e1be-904d-40d8-8f33-df320987b7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj" Nov 03 15:24:16.850191 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:16.849925 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f383e1be-904d-40d8-8f33-df320987b7cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76f64878-psglj\" (UID: \"f383e1be-904d-40d8-8f33-df320987b7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj" Nov 03 15:24:16.858576 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:16.858496 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:24:16.862013 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:16.861987 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj"] Nov 03 15:24:16.908746 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:16.908722 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:24:16.950842 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:16.950813 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f383e1be-904d-40d8-8f33-df320987b7cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76f64878-psglj\" (UID: \"f383e1be-904d-40d8-8f33-df320987b7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj" Nov 03 15:24:16.950995 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:16.950874 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f383e1be-904d-40d8-8f33-df320987b7cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76f64878-psglj\" (UID: \"f383e1be-904d-40d8-8f33-df320987b7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj" Nov 03 15:24:16.950995 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:16.950916 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vn9g\" (UniqueName: \"kubernetes.io/projected/f383e1be-904d-40d8-8f33-df320987b7cb-kube-api-access-9vn9g\") pod \"maas-api-revoke-sub-76f64878-psglj\" (UID: \"f383e1be-904d-40d8-8f33-df320987b7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj" Nov 03 15:24:16.951556 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:16.951533 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f383e1be-904d-40d8-8f33-df320987b7cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76f64878-psglj\" (UID: \"f383e1be-904d-40d8-8f33-df320987b7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj" Nov 03 15:24:16.953227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:16.953200 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f383e1be-904d-40d8-8f33-df320987b7cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76f64878-psglj\" (UID: \"f383e1be-904d-40d8-8f33-df320987b7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj" Nov 03 15:24:17.026782 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:17.026744 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vn9g\" (UniqueName: \"kubernetes.io/projected/f383e1be-904d-40d8-8f33-df320987b7cb-kube-api-access-9vn9g\") pod \"maas-api-revoke-sub-76f64878-psglj\" (UID: \"f383e1be-904d-40d8-8f33-df320987b7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj" Nov 03 15:24:17.116677 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:17.116572 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj" Nov 03 15:24:17.501082 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:17.500994 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj"] Nov 03 15:24:18.444772 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:18.444734 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj" event={"ID":"f383e1be-904d-40d8-8f33-df320987b7cb","Type":"ContainerStarted","Data":"a3c0d827248f2677ad15f87d8f5b07e619eac01f906b9bb0bb8923f5f364c9ba"} Nov 03 15:24:18.445133 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:18.444806 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj" event={"ID":"f383e1be-904d-40d8-8f33-df320987b7cb","Type":"ContainerStarted","Data":"7bb00d76aa5b5c31fd7836a6518f7c92f6268d6c74815541881c5697f2036756"} Nov 03 15:24:19.451083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:19.451047 4557 generic.go:356] "Generic (PLEG): container finished" podID="f383e1be-904d-40d8-8f33-df320987b7cb" containerID="a3c0d827248f2677ad15f87d8f5b07e619eac01f906b9bb0bb8923f5f364c9ba" exitCode=0 Nov 03 15:24:19.451487 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:19.451117 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj" event={"ID":"f383e1be-904d-40d8-8f33-df320987b7cb","Type":"ContainerDied","Data":"a3c0d827248f2677ad15f87d8f5b07e619eac01f906b9bb0bb8923f5f364c9ba"} Nov 03 15:24:19.584667 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:19.584641 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj" Nov 03 15:24:19.677878 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:19.677843 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f383e1be-904d-40d8-8f33-df320987b7cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f383e1be-904d-40d8-8f33-df320987b7cb-maas-api-openshift-ca\") pod \"f383e1be-904d-40d8-8f33-df320987b7cb\" (UID: \"f383e1be-904d-40d8-8f33-df320987b7cb\") " Nov 03 15:24:19.678017 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:19.677893 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f383e1be-904d-40d8-8f33-df320987b7cb-kube-api-access-9vn9g\" (UniqueName: \"kubernetes.io/projected/f383e1be-904d-40d8-8f33-df320987b7cb-kube-api-access-9vn9g\") pod \"f383e1be-904d-40d8-8f33-df320987b7cb\" (UID: \"f383e1be-904d-40d8-8f33-df320987b7cb\") " Nov 03 15:24:19.678017 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:19.677996 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f383e1be-904d-40d8-8f33-df320987b7cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f383e1be-904d-40d8-8f33-df320987b7cb-maas-api-xks-ca\") pod \"f383e1be-904d-40d8-8f33-df320987b7cb\" (UID: \"f383e1be-904d-40d8-8f33-df320987b7cb\") " Nov 03 15:24:19.678261 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:19.678233 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f383e1be-904d-40d8-8f33-df320987b7cb-maas-api-openshift-ca" pod "f383e1be-904d-40d8-8f33-df320987b7cb" (UID: "f383e1be-904d-40d8-8f33-df320987b7cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:24:19.678391 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:19.678338 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f383e1be-904d-40d8-8f33-df320987b7cb-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:19.679917 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:19.679891 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f383e1be-904d-40d8-8f33-df320987b7cb-kube-api-access-9vn9g" pod "f383e1be-904d-40d8-8f33-df320987b7cb" (UID: "f383e1be-904d-40d8-8f33-df320987b7cb"). InnerVolumeSpecName "kube-api-access-9vn9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:24:19.680014 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:19.679967 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f383e1be-904d-40d8-8f33-df320987b7cb-maas-api-xks-ca" pod "f383e1be-904d-40d8-8f33-df320987b7cb" (UID: "f383e1be-904d-40d8-8f33-df320987b7cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:24:19.779378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:19.779302 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vn9g\" (UniqueName: \"kubernetes.io/projected/f383e1be-904d-40d8-8f33-df320987b7cb-kube-api-access-9vn9g\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:19.779378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:19.779330 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f383e1be-904d-40d8-8f33-df320987b7cb-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:20.457277 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:20.457249 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj" Nov 03 15:24:20.457663 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:20.457283 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj" event={"ID":"f383e1be-904d-40d8-8f33-df320987b7cb","Type":"ContainerDied","Data":"7bb00d76aa5b5c31fd7836a6518f7c92f6268d6c74815541881c5697f2036756"} Nov 03 15:24:20.457663 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:20.457329 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bb00d76aa5b5c31fd7836a6518f7c92f6268d6c74815541881c5697f2036756" Nov 03 15:24:20.709882 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:20.709842 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:23.820038 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:23.820004 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:24.026126 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.026092 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46"] Nov 03 15:24:24.026519 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.026507 4557 cpu_manager.go:414] "containerMap: removing container" podUID="f383e1be-904d-40d8-8f33-df320987b7cb" containerName="revoke-keys" Nov 03 15:24:24.026566 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.026520 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f383e1be-904d-40d8-8f33-df320987b7cb" containerName="revoke-keys" Nov 03 15:24:24.026626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.026618 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="f383e1be-904d-40d8-8f33-df320987b7cb" containerName="revoke-keys" Nov 03 15:24:24.029891 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.029875 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46" Nov 03 15:24:24.086000 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.085861 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:24:24.089283 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.089224 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46"] Nov 03 15:24:24.120464 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.120432 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f11ee03-17b1-433c-810b-3672a025a470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e80bd18-8dw46\" (UID: \"3f11ee03-17b1-433c-810b-3672a025a470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46" Nov 03 15:24:24.120665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.120516 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f11ee03-17b1-433c-810b-3672a025a470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e80bd18-8dw46\" (UID: \"3f11ee03-17b1-433c-810b-3672a025a470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46" Nov 03 15:24:24.120746 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.120654 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkx85\" (UniqueName: \"kubernetes.io/projected/3f11ee03-17b1-433c-810b-3672a025a470-kube-api-access-wkx85\") pod \"maas-api-revoke-sub-8e80bd18-8dw46\" (UID: \"3f11ee03-17b1-433c-810b-3672a025a470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46" Nov 03 15:24:24.135646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.135620 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:24:24.221781 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.221754 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f11ee03-17b1-433c-810b-3672a025a470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e80bd18-8dw46\" (UID: \"3f11ee03-17b1-433c-810b-3672a025a470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46" Nov 03 15:24:24.221939 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.221801 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkx85\" (UniqueName: \"kubernetes.io/projected/3f11ee03-17b1-433c-810b-3672a025a470-kube-api-access-wkx85\") pod \"maas-api-revoke-sub-8e80bd18-8dw46\" (UID: \"3f11ee03-17b1-433c-810b-3672a025a470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46" Nov 03 15:24:24.221939 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.221828 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f11ee03-17b1-433c-810b-3672a025a470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e80bd18-8dw46\" (UID: \"3f11ee03-17b1-433c-810b-3672a025a470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46" Nov 03 15:24:24.222374 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.222354 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f11ee03-17b1-433c-810b-3672a025a470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e80bd18-8dw46\" (UID: \"3f11ee03-17b1-433c-810b-3672a025a470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46" Nov 03 15:24:24.224105 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.224087 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f11ee03-17b1-433c-810b-3672a025a470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e80bd18-8dw46\" (UID: \"3f11ee03-17b1-433c-810b-3672a025a470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46" Nov 03 15:24:24.284171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.284139 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkx85\" (UniqueName: \"kubernetes.io/projected/3f11ee03-17b1-433c-810b-3672a025a470-kube-api-access-wkx85\") pod \"maas-api-revoke-sub-8e80bd18-8dw46\" (UID: \"3f11ee03-17b1-433c-810b-3672a025a470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46" Nov 03 15:24:24.342999 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.342932 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46" Nov 03 15:24:24.717892 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:24.717860 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46"] Nov 03 15:24:24.720121 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:24:24.720092 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f11ee03_17b1_433c_810b_3672a025a470.slice/crio-cff32b5e8aed86b870c9ca7133166473b81a46861da031a82b6e1d8ebaeb80dc WatchSource:0}: Error finding container cff32b5e8aed86b870c9ca7133166473b81a46861da031a82b6e1d8ebaeb80dc: Status 404 returned error can't find the container with id cff32b5e8aed86b870c9ca7133166473b81a46861da031a82b6e1d8ebaeb80dc Nov 03 15:24:25.482897 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:25.482862 4557 generic.go:356] "Generic (PLEG): container finished" podID="3f11ee03-17b1-433c-810b-3672a025a470" containerID="406b7974f5f26cf5e2feef0ec572a56f705ad1ec4184dd22777a71ab4bac3fac" exitCode=0 Nov 03 15:24:25.483284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:25.482947 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46" event={"ID":"3f11ee03-17b1-433c-810b-3672a025a470","Type":"ContainerDied","Data":"406b7974f5f26cf5e2feef0ec572a56f705ad1ec4184dd22777a71ab4bac3fac"} Nov 03 15:24:25.483284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:25.482992 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46" event={"ID":"3f11ee03-17b1-433c-810b-3672a025a470","Type":"ContainerStarted","Data":"cff32b5e8aed86b870c9ca7133166473b81a46861da031a82b6e1d8ebaeb80dc"} Nov 03 15:24:25.617303 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:25.617271 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:26.626852 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:26.626825 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46" Nov 03 15:24:26.748912 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:26.748828 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f11ee03-17b1-433c-810b-3672a025a470-kube-api-access-wkx85\" (UniqueName: \"kubernetes.io/projected/3f11ee03-17b1-433c-810b-3672a025a470-kube-api-access-wkx85\") pod \"3f11ee03-17b1-433c-810b-3672a025a470\" (UID: \"3f11ee03-17b1-433c-810b-3672a025a470\") " Nov 03 15:24:26.748912 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:26.748899 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f11ee03-17b1-433c-810b-3672a025a470-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f11ee03-17b1-433c-810b-3672a025a470-maas-api-openshift-ca\") pod \"3f11ee03-17b1-433c-810b-3672a025a470\" (UID: \"3f11ee03-17b1-433c-810b-3672a025a470\") " Nov 03 15:24:26.749126 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:26.748978 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f11ee03-17b1-433c-810b-3672a025a470-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f11ee03-17b1-433c-810b-3672a025a470-maas-api-xks-ca\") pod \"3f11ee03-17b1-433c-810b-3672a025a470\" (UID: \"3f11ee03-17b1-433c-810b-3672a025a470\") " Nov 03 15:24:26.749281 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:26.749259 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f11ee03-17b1-433c-810b-3672a025a470-maas-api-openshift-ca" pod "3f11ee03-17b1-433c-810b-3672a025a470" (UID: "3f11ee03-17b1-433c-810b-3672a025a470"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:24:26.749379 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:26.749340 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f11ee03-17b1-433c-810b-3672a025a470-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:26.751332 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:26.751300 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f11ee03-17b1-433c-810b-3672a025a470-maas-api-xks-ca" pod "3f11ee03-17b1-433c-810b-3672a025a470" (UID: "3f11ee03-17b1-433c-810b-3672a025a470"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:24:26.751544 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:26.751523 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f11ee03-17b1-433c-810b-3672a025a470-kube-api-access-wkx85" pod "3f11ee03-17b1-433c-810b-3672a025a470" (UID: "3f11ee03-17b1-433c-810b-3672a025a470"). InnerVolumeSpecName "kube-api-access-wkx85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:24:26.850156 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:26.850129 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f11ee03-17b1-433c-810b-3672a025a470-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:26.850156 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:26.850153 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkx85\" (UniqueName: \"kubernetes.io/projected/3f11ee03-17b1-433c-810b-3672a025a470-kube-api-access-wkx85\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:27.495572 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:27.495545 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46" Nov 03 15:24:27.495572 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:27.495561 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46" event={"ID":"3f11ee03-17b1-433c-810b-3672a025a470","Type":"ContainerDied","Data":"cff32b5e8aed86b870c9ca7133166473b81a46861da031a82b6e1d8ebaeb80dc"} Nov 03 15:24:27.495858 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:27.495633 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cff32b5e8aed86b870c9ca7133166473b81a46861da031a82b6e1d8ebaeb80dc" Nov 03 15:24:28.521464 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.521423 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4"] Nov 03 15:24:28.521911 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.521885 4557 cpu_manager.go:414] "containerMap: removing container" podUID="3f11ee03-17b1-433c-810b-3672a025a470" containerName="revoke-keys" Nov 03 15:24:28.521911 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.521898 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f11ee03-17b1-433c-810b-3672a025a470" containerName="revoke-keys" Nov 03 15:24:28.521984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.521968 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f11ee03-17b1-433c-810b-3672a025a470" containerName="revoke-keys" Nov 03 15:24:28.524875 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.524856 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.581258 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.581232 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-bc1f07-openshift-default-dockercfg-lrpxt\"" Nov 03 15:24:28.633378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.633346 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4"] Nov 03 15:24:28.668035 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.667999 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.668204 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.668044 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.668204 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.668089 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.668289 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.668218 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.668289 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.668249 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv7r5\" (UniqueName: \"kubernetes.io/projected/34f61e10-89c2-4c65-be7d-31a468a7f7e6-kube-api-access-cv7r5\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.668289 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.668271 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.668428 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.668330 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.668428 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.668392 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.668789 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.668753 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.769991 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.769959 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.769991 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.769993 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv7r5\" (UniqueName: \"kubernetes.io/projected/34f61e10-89c2-4c65-be7d-31a468a7f7e6-kube-api-access-cv7r5\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.770193 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.770013 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.770193 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.770124 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.770193 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.770188 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.770369 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.770345 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.770477 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.770459 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.770555 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.770485 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.770555 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.770504 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.770688 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.770559 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.770688 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.770655 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.770789 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.770739 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.770789 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.770749 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.770895 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.770873 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.772532 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.772470 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.772842 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.772821 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.833267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.833234 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv7r5\" (UniqueName: \"kubernetes.io/projected/34f61e10-89c2-4c65-be7d-31a468a7f7e6-kube-api-access-cv7r5\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.834337 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.834310 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:28.837125 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:28.837101 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:29.301281 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.301254 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4"] Nov 03 15:24:29.301941 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:24:29.301919 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f61e10_89c2_4c65_be7d_31a468a7f7e6.slice/crio-ff4419c13b322d3d111d29a9960b6d4c228adf9c24acdb473d37af2bf08c26e5 WatchSource:0}: Error finding container ff4419c13b322d3d111d29a9960b6d4c228adf9c24acdb473d37af2bf08c26e5: Status 404 returned error can't find the container with id ff4419c13b322d3d111d29a9960b6d4c228adf9c24acdb473d37af2bf08c26e5 Nov 03 15:24:29.303672 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.303634 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4"] Nov 03 15:24:29.304403 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.304377 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:24:29.304460 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.304435 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:24:29.304500 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.304464 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:24:29.308432 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.308413 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4" Nov 03 15:24:29.362811 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.362787 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:24:29.362998 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.362984 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:24:29.375946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.375911 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-596be1cd-kdgk4\" (UID: \"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4" Nov 03 15:24:29.376124 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.376106 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz6dg\" (UniqueName: \"kubernetes.io/projected/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-kube-api-access-wz6dg\") pod \"maas-api-revoke-sub-596be1cd-kdgk4\" (UID: \"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4" Nov 03 15:24:29.376287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.376261 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-596be1cd-kdgk4\" (UID: \"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4" Nov 03 15:24:29.390133 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.390109 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4"] Nov 03 15:24:29.477050 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.477012 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz6dg\" (UniqueName: \"kubernetes.io/projected/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-kube-api-access-wz6dg\") pod \"maas-api-revoke-sub-596be1cd-kdgk4\" (UID: \"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4" Nov 03 15:24:29.477210 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.477066 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-596be1cd-kdgk4\" (UID: \"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4" Nov 03 15:24:29.477210 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.477113 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-596be1cd-kdgk4\" (UID: \"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4" Nov 03 15:24:29.477764 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.477745 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-596be1cd-kdgk4\" (UID: \"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4" Nov 03 15:24:29.479286 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.479270 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-596be1cd-kdgk4\" (UID: \"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4" Nov 03 15:24:29.516055 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.516016 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" event={"ID":"34f61e10-89c2-4c65-be7d-31a468a7f7e6","Type":"ContainerStarted","Data":"6c1cb72ff273f054004fe29975e0fb3f0f7243c1e09155adbd850558622f3247"} Nov 03 15:24:29.516206 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.516074 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" event={"ID":"34f61e10-89c2-4c65-be7d-31a468a7f7e6","Type":"ContainerStarted","Data":"ff4419c13b322d3d111d29a9960b6d4c228adf9c24acdb473d37af2bf08c26e5"} Nov 03 15:24:29.539564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.539525 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz6dg\" (UniqueName: \"kubernetes.io/projected/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-kube-api-access-wz6dg\") pod \"maas-api-revoke-sub-596be1cd-kdgk4\" (UID: \"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4" Nov 03 15:24:29.633298 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.633214 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4" Nov 03 15:24:29.641177 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.641118 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" podStartSLOduration=1.641100869 podStartE2EDuration="1.641100869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:24:29.640348334 +0000 UTC m=+979.483469879" watchObservedRunningTime="2026-11-03 15:24:29.641100869 +0000 UTC m=+979.484222412" Nov 03 15:24:29.837738 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:29.837708 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:30.019158 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:30.019133 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4"] Nov 03 15:24:30.021381 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:24:30.021358 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3ed764_4fb3_46b1_88b6_1dc1dcff3154.slice/crio-2770dc50367b83d4f9be3d290b138d08d08d2ef60217344790c5c30d18fd29be WatchSource:0}: Error finding container 2770dc50367b83d4f9be3d290b138d08d08d2ef60217344790c5c30d18fd29be: Status 404 returned error can't find the container with id 2770dc50367b83d4f9be3d290b138d08d08d2ef60217344790c5c30d18fd29be Nov 03 15:24:30.519626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:30.519471 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:30.533967 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:30.533933 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4" event={"ID":"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154","Type":"ContainerStarted","Data":"160172312d88cdb8f7eeb2c5585b3f84bb82d3baae7380562c43e529d17d9b49"} Nov 03 15:24:30.534132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:30.534018 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4" event={"ID":"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154","Type":"ContainerStarted","Data":"2770dc50367b83d4f9be3d290b138d08d08d2ef60217344790c5c30d18fd29be"} Nov 03 15:24:30.659633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:30.659517 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4" podStartSLOduration=1.65950469 podStartE2EDuration="1.65950469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:24:30.657728076 +0000 UTC m=+980.500849617" watchObservedRunningTime="2026-11-03 15:24:30.65950469 +0000 UTC m=+980.502626231" Nov 03 15:24:30.839135 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:30.839099 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 15:24:30.839316 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:30.839162 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" podUID="34f61e10-89c2-4c65-be7d-31a468a7f7e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 15:24:31.540863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:31.540778 4557 generic.go:356] "Generic (PLEG): container finished" podID="2f3ed764-4fb3-46b1-88b6-1dc1dcff3154" containerID="160172312d88cdb8f7eeb2c5585b3f84bb82d3baae7380562c43e529d17d9b49" exitCode=0 Nov 03 15:24:31.541006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:31.540862 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4" event={"ID":"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154","Type":"ContainerDied","Data":"160172312d88cdb8f7eeb2c5585b3f84bb82d3baae7380562c43e529d17d9b49"} Nov 03 15:24:31.766644 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:31.766606 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:31.837433 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:31.837399 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 15:24:31.837635 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:31.837451 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" podUID="34f61e10-89c2-4c65-be7d-31a468a7f7e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 15:24:31.971181 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:31.971133 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:32.713613 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:32.713571 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4" Nov 03 15:24:32.829109 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:32.829026 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-maas-api-xks-ca\") pod \"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154\" (UID: \"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154\") " Nov 03 15:24:32.829109 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:32.829081 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-maas-api-openshift-ca\") pod \"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154\" (UID: \"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154\") " Nov 03 15:24:32.829573 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:32.829121 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-kube-api-access-wz6dg\" (UniqueName: \"kubernetes.io/projected/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-kube-api-access-wz6dg\") pod \"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154\" (UID: \"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154\") " Nov 03 15:24:32.829573 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:32.829390 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-maas-api-openshift-ca" pod "2f3ed764-4fb3-46b1-88b6-1dc1dcff3154" (UID: "2f3ed764-4fb3-46b1-88b6-1dc1dcff3154"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:24:32.829573 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:32.829557 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:32.831136 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:32.831109 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-maas-api-xks-ca" pod "2f3ed764-4fb3-46b1-88b6-1dc1dcff3154" (UID: "2f3ed764-4fb3-46b1-88b6-1dc1dcff3154"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:24:32.831245 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:32.831166 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-kube-api-access-wz6dg" pod "2f3ed764-4fb3-46b1-88b6-1dc1dcff3154" (UID: "2f3ed764-4fb3-46b1-88b6-1dc1dcff3154"). InnerVolumeSpecName "kube-api-access-wz6dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:24:32.837760 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:32.837734 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 15:24:32.837888 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:32.837802 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" podUID="34f61e10-89c2-4c65-be7d-31a468a7f7e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 15:24:32.930186 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:32.930148 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:32.930186 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:32.930184 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz6dg\" (UniqueName: \"kubernetes.io/projected/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154-kube-api-access-wz6dg\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:33.551778 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:33.551751 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4" Nov 03 15:24:33.551935 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:33.551794 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4" event={"ID":"2f3ed764-4fb3-46b1-88b6-1dc1dcff3154","Type":"ContainerDied","Data":"2770dc50367b83d4f9be3d290b138d08d08d2ef60217344790c5c30d18fd29be"} Nov 03 15:24:33.551935 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:33.551839 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2770dc50367b83d4f9be3d290b138d08d08d2ef60217344790c5c30d18fd29be" Nov 03 15:24:33.837510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:33.837428 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 15:24:33.837510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:33.837481 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" podUID="34f61e10-89c2-4c65-be7d-31a468a7f7e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 15:24:34.838106 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:34.838074 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 15:24:34.838477 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:34.838131 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" podUID="34f61e10-89c2-4c65-be7d-31a468a7f7e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 15:24:35.497127 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:35.497044 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:35.838159 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:35.838109 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 15:24:35.838705 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:35.838188 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" podUID="34f61e10-89c2-4c65-be7d-31a468a7f7e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 15:24:36.844389 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:36.844352 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:36.844940 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:36.844750 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:36.845478 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:36.845460 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:24:38.008375 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.008341 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn"] Nov 03 15:24:38.008792 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.008719 4557 cpu_manager.go:414] "containerMap: removing container" podUID="2f3ed764-4fb3-46b1-88b6-1dc1dcff3154" containerName="revoke-keys" Nov 03 15:24:38.008792 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.008729 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3ed764-4fb3-46b1-88b6-1dc1dcff3154" containerName="revoke-keys" Nov 03 15:24:38.008865 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.008833 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3ed764-4fb3-46b1-88b6-1dc1dcff3154" containerName="revoke-keys" Nov 03 15:24:38.012064 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.012048 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn" Nov 03 15:24:38.067671 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.067446 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:24:38.083982 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.083950 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-732560b8-9hxpn\" (UID: \"eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn" Nov 03 15:24:38.084172 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.083993 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-732560b8-9hxpn\" (UID: \"eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn" Nov 03 15:24:38.084172 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.084146 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr9j8\" (UniqueName: \"kubernetes.io/projected/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-kube-api-access-fr9j8\") pod \"maas-api-revoke-sub-732560b8-9hxpn\" (UID: \"eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn" Nov 03 15:24:38.117315 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.117284 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:24:38.119707 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.119686 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn"] Nov 03 15:24:38.184632 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.184572 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-732560b8-9hxpn\" (UID: \"eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn" Nov 03 15:24:38.184815 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.184645 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-732560b8-9hxpn\" (UID: \"eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn" Nov 03 15:24:38.184815 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.184794 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr9j8\" (UniqueName: \"kubernetes.io/projected/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-kube-api-access-fr9j8\") pod \"maas-api-revoke-sub-732560b8-9hxpn\" (UID: \"eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn" Nov 03 15:24:38.185312 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.185292 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-732560b8-9hxpn\" (UID: \"eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn" Nov 03 15:24:38.186923 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.186899 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-732560b8-9hxpn\" (UID: \"eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn" Nov 03 15:24:38.245866 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.245844 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr9j8\" (UniqueName: \"kubernetes.io/projected/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-kube-api-access-fr9j8\") pod \"maas-api-revoke-sub-732560b8-9hxpn\" (UID: \"eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn" Nov 03 15:24:38.325455 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.325429 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn" Nov 03 15:24:38.701136 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:38.701110 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn"] Nov 03 15:24:38.702407 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:24:38.702384 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeee83d4_4b43_4cc1_8c9c_85b5b32fe57b.slice/crio-eb5eec5031a30f5f8705042b79e07127b754993d51395314afb2485181acb8d8 WatchSource:0}: Error finding container eb5eec5031a30f5f8705042b79e07127b754993d51395314afb2485181acb8d8: Status 404 returned error can't find the container with id eb5eec5031a30f5f8705042b79e07127b754993d51395314afb2485181acb8d8 Nov 03 15:24:39.587963 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:39.587932 4557 generic.go:356] "Generic (PLEG): container finished" podID="eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b" containerID="a49e50c9ef4d8aa52aa032fc4857136b2c8df3187a4806c92a70f32b400d2eb5" exitCode=0 Nov 03 15:24:39.588286 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:39.588011 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn" event={"ID":"eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b","Type":"ContainerDied","Data":"a49e50c9ef4d8aa52aa032fc4857136b2c8df3187a4806c92a70f32b400d2eb5"} Nov 03 15:24:39.588286 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:39.588057 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn" event={"ID":"eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b","Type":"ContainerStarted","Data":"eb5eec5031a30f5f8705042b79e07127b754993d51395314afb2485181acb8d8"} Nov 03 15:24:40.418106 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:40.418067 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:40.736403 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:40.736370 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn" Nov 03 15:24:40.805565 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:40.805528 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-kube-api-access-fr9j8\" (UniqueName: \"kubernetes.io/projected/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-kube-api-access-fr9j8\") pod \"eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b\" (UID: \"eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b\") " Nov 03 15:24:40.805768 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:40.805613 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-maas-api-openshift-ca\") pod \"eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b\" (UID: \"eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b\") " Nov 03 15:24:40.805768 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:40.805724 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-maas-api-xks-ca\") pod \"eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b\" (UID: \"eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b\") " Nov 03 15:24:40.806062 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:40.806033 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-maas-api-openshift-ca" pod "eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b" (UID: "eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:24:40.806156 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:40.806142 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:40.807704 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:40.807682 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-maas-api-xks-ca" pod "eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b" (UID: "eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:24:40.807817 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:40.807712 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-kube-api-access-fr9j8" pod "eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b" (UID: "eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b"). InnerVolumeSpecName "kube-api-access-fr9j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:24:40.907460 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:40.907424 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr9j8\" (UniqueName: \"kubernetes.io/projected/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-kube-api-access-fr9j8\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:40.907460 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:40.907455 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:41.067049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.067009 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8"] Nov 03 15:24:41.067458 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.067443 4557 cpu_manager.go:414] "containerMap: removing container" podUID="eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b" containerName="revoke-keys" Nov 03 15:24:41.067506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.067459 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b" containerName="revoke-keys" Nov 03 15:24:41.067561 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.067554 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b" containerName="revoke-keys" Nov 03 15:24:41.070896 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.070874 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8" Nov 03 15:24:41.110157 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.110126 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c780ce0-9ptc8\" (UID: \"a9b2a25f-7307-41c6-9fee-902e24f9ac3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8" Nov 03 15:24:41.110317 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.110172 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcnrn\" (UniqueName: \"kubernetes.io/projected/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-kube-api-access-xcnrn\") pod \"maas-api-revoke-sub-9c780ce0-9ptc8\" (UID: \"a9b2a25f-7307-41c6-9fee-902e24f9ac3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8" Nov 03 15:24:41.110398 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.110315 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c780ce0-9ptc8\" (UID: \"a9b2a25f-7307-41c6-9fee-902e24f9ac3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8" Nov 03 15:24:41.129474 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.129433 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8"] Nov 03 15:24:41.211944 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.211896 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c780ce0-9ptc8\" (UID: \"a9b2a25f-7307-41c6-9fee-902e24f9ac3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8" Nov 03 15:24:41.212137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.211994 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c780ce0-9ptc8\" (UID: \"a9b2a25f-7307-41c6-9fee-902e24f9ac3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8" Nov 03 15:24:41.212137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.212024 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcnrn\" (UniqueName: \"kubernetes.io/projected/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-kube-api-access-xcnrn\") pod \"maas-api-revoke-sub-9c780ce0-9ptc8\" (UID: \"a9b2a25f-7307-41c6-9fee-902e24f9ac3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8" Nov 03 15:24:41.212607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.212560 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c780ce0-9ptc8\" (UID: \"a9b2a25f-7307-41c6-9fee-902e24f9ac3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8" Nov 03 15:24:41.214291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.214269 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c780ce0-9ptc8\" (UID: \"a9b2a25f-7307-41c6-9fee-902e24f9ac3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8" Nov 03 15:24:41.274049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.274018 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcnrn\" (UniqueName: \"kubernetes.io/projected/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-kube-api-access-xcnrn\") pod \"maas-api-revoke-sub-9c780ce0-9ptc8\" (UID: \"a9b2a25f-7307-41c6-9fee-902e24f9ac3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8" Nov 03 15:24:41.385658 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.385559 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8" Nov 03 15:24:41.600455 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.600419 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn" event={"ID":"eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b","Type":"ContainerDied","Data":"eb5eec5031a30f5f8705042b79e07127b754993d51395314afb2485181acb8d8"} Nov 03 15:24:41.600455 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.600447 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn" Nov 03 15:24:41.600695 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.600469 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb5eec5031a30f5f8705042b79e07127b754993d51395314afb2485181acb8d8" Nov 03 15:24:41.760424 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:41.760393 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8"] Nov 03 15:24:41.762311 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:24:41.762287 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b2a25f_7307_41c6_9fee_902e24f9ac3b.slice/crio-267108fe488b4f86b21796075552e9a3f93a5b74c0c4ed2b935a565d5f45cb93 WatchSource:0}: Error finding container 267108fe488b4f86b21796075552e9a3f93a5b74c0c4ed2b935a565d5f45cb93: Status 404 returned error can't find the container with id 267108fe488b4f86b21796075552e9a3f93a5b74c0c4ed2b935a565d5f45cb93 Nov 03 15:24:42.608228 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:42.608181 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8" event={"ID":"a9b2a25f-7307-41c6-9fee-902e24f9ac3b","Type":"ContainerStarted","Data":"490a492ee33e82a91261cb56dd78e064467691262d0d1d0119881176ea715aeb"} Nov 03 15:24:42.608415 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:42.608269 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8" event={"ID":"a9b2a25f-7307-41c6-9fee-902e24f9ac3b","Type":"ContainerStarted","Data":"267108fe488b4f86b21796075552e9a3f93a5b74c0c4ed2b935a565d5f45cb93"} Nov 03 15:24:42.725931 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:42.725870 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8" podStartSLOduration=1.7258535419999999 podStartE2EDuration="1.725853542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:24:42.723914597 +0000 UTC m=+992.567036139" watchObservedRunningTime="2026-11-03 15:24:42.725853542 +0000 UTC m=+992.568975091" Nov 03 15:24:43.616662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:43.616625 4557 generic.go:356] "Generic (PLEG): container finished" podID="a9b2a25f-7307-41c6-9fee-902e24f9ac3b" containerID="490a492ee33e82a91261cb56dd78e064467691262d0d1d0119881176ea715aeb" exitCode=0 Nov 03 15:24:43.617058 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:43.616673 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8" event={"ID":"a9b2a25f-7307-41c6-9fee-902e24f9ac3b","Type":"ContainerDied","Data":"490a492ee33e82a91261cb56dd78e064467691262d0d1d0119881176ea715aeb"} Nov 03 15:24:43.708457 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:43.708423 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:43.877971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:43.877892 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7"] Nov 03 15:24:43.881557 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:43.881536 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7" Nov 03 15:24:43.947812 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:43.947432 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec209e9-6c77-4d08-801a-7195b980e41e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b45612a-sklh7\" (UID: \"6ec209e9-6c77-4d08-801a-7195b980e41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7" Nov 03 15:24:43.947812 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:43.947504 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec209e9-6c77-4d08-801a-7195b980e41e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b45612a-sklh7\" (UID: \"6ec209e9-6c77-4d08-801a-7195b980e41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7" Nov 03 15:24:43.947812 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:43.947661 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xmgz\" (UniqueName: \"kubernetes.io/projected/6ec209e9-6c77-4d08-801a-7195b980e41e-kube-api-access-5xmgz\") pod \"maas-api-revoke-sub-9b45612a-sklh7\" (UID: \"6ec209e9-6c77-4d08-801a-7195b980e41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7" Nov 03 15:24:43.948226 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:43.948153 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7"] Nov 03 15:24:44.049171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.049137 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec209e9-6c77-4d08-801a-7195b980e41e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b45612a-sklh7\" (UID: \"6ec209e9-6c77-4d08-801a-7195b980e41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7" Nov 03 15:24:44.049320 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.049181 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec209e9-6c77-4d08-801a-7195b980e41e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b45612a-sklh7\" (UID: \"6ec209e9-6c77-4d08-801a-7195b980e41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7" Nov 03 15:24:44.049320 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.049273 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xmgz\" (UniqueName: \"kubernetes.io/projected/6ec209e9-6c77-4d08-801a-7195b980e41e-kube-api-access-5xmgz\") pod \"maas-api-revoke-sub-9b45612a-sklh7\" (UID: \"6ec209e9-6c77-4d08-801a-7195b980e41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7" Nov 03 15:24:44.049783 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.049763 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec209e9-6c77-4d08-801a-7195b980e41e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b45612a-sklh7\" (UID: \"6ec209e9-6c77-4d08-801a-7195b980e41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7" Nov 03 15:24:44.051518 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.051495 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec209e9-6c77-4d08-801a-7195b980e41e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b45612a-sklh7\" (UID: \"6ec209e9-6c77-4d08-801a-7195b980e41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7" Nov 03 15:24:44.110625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.110593 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xmgz\" (UniqueName: \"kubernetes.io/projected/6ec209e9-6c77-4d08-801a-7195b980e41e-kube-api-access-5xmgz\") pod \"maas-api-revoke-sub-9b45612a-sklh7\" (UID: \"6ec209e9-6c77-4d08-801a-7195b980e41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7" Nov 03 15:24:44.194568 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.194489 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7" Nov 03 15:24:44.567342 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.567311 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7"] Nov 03 15:24:44.568337 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:24:44.568312 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec209e9_6c77_4d08_801a_7195b980e41e.slice/crio-f1a75a2325602885c221b2701a042e4970dc380f8d4096c83d9e5f649bd68594 WatchSource:0}: Error finding container f1a75a2325602885c221b2701a042e4970dc380f8d4096c83d9e5f649bd68594: Status 404 returned error can't find the container with id f1a75a2325602885c221b2701a042e4970dc380f8d4096c83d9e5f649bd68594 Nov 03 15:24:44.623007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.622976 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7" event={"ID":"6ec209e9-6c77-4d08-801a-7195b980e41e","Type":"ContainerStarted","Data":"f1a75a2325602885c221b2701a042e4970dc380f8d4096c83d9e5f649bd68594"} Nov 03 15:24:44.751156 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.751130 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8" Nov 03 15:24:44.858246 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.858176 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-maas-api-xks-ca\") pod \"a9b2a25f-7307-41c6-9fee-902e24f9ac3b\" (UID: \"a9b2a25f-7307-41c6-9fee-902e24f9ac3b\") " Nov 03 15:24:44.858387 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.858297 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-kube-api-access-xcnrn\" (UniqueName: \"kubernetes.io/projected/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-kube-api-access-xcnrn\") pod \"a9b2a25f-7307-41c6-9fee-902e24f9ac3b\" (UID: \"a9b2a25f-7307-41c6-9fee-902e24f9ac3b\") " Nov 03 15:24:44.858387 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.858319 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-maas-api-openshift-ca\") pod \"a9b2a25f-7307-41c6-9fee-902e24f9ac3b\" (UID: \"a9b2a25f-7307-41c6-9fee-902e24f9ac3b\") " Nov 03 15:24:44.858759 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.858731 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-maas-api-openshift-ca" pod "a9b2a25f-7307-41c6-9fee-902e24f9ac3b" (UID: "a9b2a25f-7307-41c6-9fee-902e24f9ac3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:24:44.860155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.860134 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-maas-api-xks-ca" pod "a9b2a25f-7307-41c6-9fee-902e24f9ac3b" (UID: "a9b2a25f-7307-41c6-9fee-902e24f9ac3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:24:44.860262 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.860242 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-kube-api-access-xcnrn" pod "a9b2a25f-7307-41c6-9fee-902e24f9ac3b" (UID: "a9b2a25f-7307-41c6-9fee-902e24f9ac3b"). InnerVolumeSpecName "kube-api-access-xcnrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:24:44.960005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.959972 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:44.960005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.960001 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcnrn\" (UniqueName: \"kubernetes.io/projected/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-kube-api-access-xcnrn\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:44.960005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:44.960011 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2a25f-7307-41c6-9fee-902e24f9ac3b-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:45.317283 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:45.317232 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:45.630550 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:45.630462 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8" Nov 03 15:24:45.630550 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:45.630477 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8" event={"ID":"a9b2a25f-7307-41c6-9fee-902e24f9ac3b","Type":"ContainerDied","Data":"267108fe488b4f86b21796075552e9a3f93a5b74c0c4ed2b935a565d5f45cb93"} Nov 03 15:24:45.630550 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:45.630530 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="267108fe488b4f86b21796075552e9a3f93a5b74c0c4ed2b935a565d5f45cb93" Nov 03 15:24:45.632008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:45.631987 4557 generic.go:356] "Generic (PLEG): container finished" podID="6ec209e9-6c77-4d08-801a-7195b980e41e" containerID="4a6b66b7f98ae4ae99070e15bc5453a3a9f51c7bf892dbb75d4cbb1ee688ac13" exitCode=0 Nov 03 15:24:45.632109 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:45.632068 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7" event={"ID":"6ec209e9-6c77-4d08-801a-7195b980e41e","Type":"ContainerDied","Data":"4a6b66b7f98ae4ae99070e15bc5453a3a9f51c7bf892dbb75d4cbb1ee688ac13"} Nov 03 15:24:46.530365 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:24:46.530329 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:24:46.549337 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.549302 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:46.742136 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.742107 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn"] Nov 03 15:24:46.742523 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.742510 4557 cpu_manager.go:414] "containerMap: removing container" podUID="a9b2a25f-7307-41c6-9fee-902e24f9ac3b" containerName="revoke-keys" Nov 03 15:24:46.742561 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.742523 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b2a25f-7307-41c6-9fee-902e24f9ac3b" containerName="revoke-keys" Nov 03 15:24:46.742619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.742604 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b2a25f-7307-41c6-9fee-902e24f9ac3b" containerName="revoke-keys" Nov 03 15:24:46.745779 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.745760 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn" Nov 03 15:24:46.776866 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.776843 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0114e22f-tmbsn\" (UID: \"ac5cb6ab-283f-4ff1-a3ea-f8d958451962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn" Nov 03 15:24:46.776962 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.776875 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5829\" (UniqueName: \"kubernetes.io/projected/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-kube-api-access-p5829\") pod \"maas-api-revoke-sub-0114e22f-tmbsn\" (UID: \"ac5cb6ab-283f-4ff1-a3ea-f8d958451962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn" Nov 03 15:24:46.776962 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.776896 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0114e22f-tmbsn\" (UID: \"ac5cb6ab-283f-4ff1-a3ea-f8d958451962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn" Nov 03 15:24:46.777429 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.777414 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7" Nov 03 15:24:46.804264 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.804237 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn"] Nov 03 15:24:46.877703 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.877674 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec209e9-6c77-4d08-801a-7195b980e41e-kube-api-access-5xmgz\" (UniqueName: \"kubernetes.io/projected/6ec209e9-6c77-4d08-801a-7195b980e41e-kube-api-access-5xmgz\") pod \"6ec209e9-6c77-4d08-801a-7195b980e41e\" (UID: \"6ec209e9-6c77-4d08-801a-7195b980e41e\") " Nov 03 15:24:46.877870 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.877757 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec209e9-6c77-4d08-801a-7195b980e41e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec209e9-6c77-4d08-801a-7195b980e41e-maas-api-openshift-ca\") pod \"6ec209e9-6c77-4d08-801a-7195b980e41e\" (UID: \"6ec209e9-6c77-4d08-801a-7195b980e41e\") " Nov 03 15:24:46.877870 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.877838 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ec209e9-6c77-4d08-801a-7195b980e41e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec209e9-6c77-4d08-801a-7195b980e41e-maas-api-xks-ca\") pod \"6ec209e9-6c77-4d08-801a-7195b980e41e\" (UID: \"6ec209e9-6c77-4d08-801a-7195b980e41e\") " Nov 03 15:24:46.877975 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.877959 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0114e22f-tmbsn\" (UID: \"ac5cb6ab-283f-4ff1-a3ea-f8d958451962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn" Nov 03 15:24:46.878040 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.877987 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5829\" (UniqueName: \"kubernetes.io/projected/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-kube-api-access-p5829\") pod \"maas-api-revoke-sub-0114e22f-tmbsn\" (UID: \"ac5cb6ab-283f-4ff1-a3ea-f8d958451962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn" Nov 03 15:24:46.878040 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.878005 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0114e22f-tmbsn\" (UID: \"ac5cb6ab-283f-4ff1-a3ea-f8d958451962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn" Nov 03 15:24:46.878142 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.878109 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec209e9-6c77-4d08-801a-7195b980e41e-maas-api-openshift-ca" pod "6ec209e9-6c77-4d08-801a-7195b980e41e" (UID: "6ec209e9-6c77-4d08-801a-7195b980e41e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:24:46.878243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.878217 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec209e9-6c77-4d08-801a-7195b980e41e-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:46.878788 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.878754 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0114e22f-tmbsn\" (UID: \"ac5cb6ab-283f-4ff1-a3ea-f8d958451962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn" Nov 03 15:24:46.880000 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.879976 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec209e9-6c77-4d08-801a-7195b980e41e-maas-api-xks-ca" pod "6ec209e9-6c77-4d08-801a-7195b980e41e" (UID: "6ec209e9-6c77-4d08-801a-7195b980e41e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:24:46.880086 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.880037 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec209e9-6c77-4d08-801a-7195b980e41e-kube-api-access-5xmgz" pod "6ec209e9-6c77-4d08-801a-7195b980e41e" (UID: "6ec209e9-6c77-4d08-801a-7195b980e41e"). InnerVolumeSpecName "kube-api-access-5xmgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:24:46.880327 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.880307 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0114e22f-tmbsn\" (UID: \"ac5cb6ab-283f-4ff1-a3ea-f8d958451962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn" Nov 03 15:24:46.939028 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.938994 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5829\" (UniqueName: \"kubernetes.io/projected/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-kube-api-access-p5829\") pod \"maas-api-revoke-sub-0114e22f-tmbsn\" (UID: \"ac5cb6ab-283f-4ff1-a3ea-f8d958451962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn" Nov 03 15:24:46.979202 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.979173 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xmgz\" (UniqueName: \"kubernetes.io/projected/6ec209e9-6c77-4d08-801a-7195b980e41e-kube-api-access-5xmgz\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:46.979338 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:46.979298 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec209e9-6c77-4d08-801a-7195b980e41e-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:47.076026 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:47.075951 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn" Nov 03 15:24:47.449110 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:47.449078 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn"] Nov 03 15:24:47.450744 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:24:47.450716 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5cb6ab_283f_4ff1_a3ea_f8d958451962.slice/crio-46c1a76c233dd0c3d5fad755bc74fbe11382fe1488de198ab8d486081b9ba557 WatchSource:0}: Error finding container 46c1a76c233dd0c3d5fad755bc74fbe11382fe1488de198ab8d486081b9ba557: Status 404 returned error can't find the container with id 46c1a76c233dd0c3d5fad755bc74fbe11382fe1488de198ab8d486081b9ba557 Nov 03 15:24:47.643885 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:47.643846 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn" event={"ID":"ac5cb6ab-283f-4ff1-a3ea-f8d958451962","Type":"ContainerStarted","Data":"46c1a76c233dd0c3d5fad755bc74fbe11382fe1488de198ab8d486081b9ba557"} Nov 03 15:24:47.645249 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:47.645225 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7" event={"ID":"6ec209e9-6c77-4d08-801a-7195b980e41e","Type":"ContainerDied","Data":"f1a75a2325602885c221b2701a042e4970dc380f8d4096c83d9e5f649bd68594"} Nov 03 15:24:47.645377 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:47.645263 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1a75a2325602885c221b2701a042e4970dc380f8d4096c83d9e5f649bd68594" Nov 03 15:24:47.645377 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:47.645262 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7" Nov 03 15:24:48.653798 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:48.653711 4557 generic.go:356] "Generic (PLEG): container finished" podID="ac5cb6ab-283f-4ff1-a3ea-f8d958451962" containerID="63bff8c04b1232c099b872b56c5d737ebc2729e132bc8cfd1082f24624fbec30" exitCode=0 Nov 03 15:24:48.654157 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:48.653791 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn" event={"ID":"ac5cb6ab-283f-4ff1-a3ea-f8d958451962","Type":"ContainerDied","Data":"63bff8c04b1232c099b872b56c5d737ebc2729e132bc8cfd1082f24624fbec30"} Nov 03 15:24:48.814227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:48.814189 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:49.004261 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:49.004181 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:49.784457 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:49.784431 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn" Nov 03 15:24:49.907194 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:49.907163 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-maas-api-openshift-ca\") pod \"ac5cb6ab-283f-4ff1-a3ea-f8d958451962\" (UID: \"ac5cb6ab-283f-4ff1-a3ea-f8d958451962\") " Nov 03 15:24:49.907348 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:49.907270 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-kube-api-access-p5829\" (UniqueName: \"kubernetes.io/projected/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-kube-api-access-p5829\") pod \"ac5cb6ab-283f-4ff1-a3ea-f8d958451962\" (UID: \"ac5cb6ab-283f-4ff1-a3ea-f8d958451962\") " Nov 03 15:24:49.907348 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:49.907307 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-maas-api-xks-ca\") pod \"ac5cb6ab-283f-4ff1-a3ea-f8d958451962\" (UID: \"ac5cb6ab-283f-4ff1-a3ea-f8d958451962\") " Nov 03 15:24:49.907647 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:49.907618 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-maas-api-openshift-ca" pod "ac5cb6ab-283f-4ff1-a3ea-f8d958451962" (UID: "ac5cb6ab-283f-4ff1-a3ea-f8d958451962"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:24:49.910773 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:49.910711 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-maas-api-xks-ca" pod "ac5cb6ab-283f-4ff1-a3ea-f8d958451962" (UID: "ac5cb6ab-283f-4ff1-a3ea-f8d958451962"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:24:49.910896 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:49.910872 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-kube-api-access-p5829" pod "ac5cb6ab-283f-4ff1-a3ea-f8d958451962" (UID: "ac5cb6ab-283f-4ff1-a3ea-f8d958451962"). InnerVolumeSpecName "kube-api-access-p5829". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:24:50.010015 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:50.009978 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5829\" (UniqueName: \"kubernetes.io/projected/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-kube-api-access-p5829\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:50.010015 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:50.010010 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:50.010015 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:50.010022 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5cb6ab-283f-4ff1-a3ea-f8d958451962-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:50.209908 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:50.209815 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:50.664301 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:50.664278 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn" Nov 03 15:24:50.664456 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:50.664307 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn" event={"ID":"ac5cb6ab-283f-4ff1-a3ea-f8d958451962","Type":"ContainerDied","Data":"46c1a76c233dd0c3d5fad755bc74fbe11382fe1488de198ab8d486081b9ba557"} Nov 03 15:24:50.664456 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:50.664354 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46c1a76c233dd0c3d5fad755bc74fbe11382fe1488de198ab8d486081b9ba557" Nov 03 15:24:55.112808 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.112776 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:24:55.310180 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.310146 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn"] Nov 03 15:24:55.310637 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.310619 4557 cpu_manager.go:414] "containerMap: removing container" podUID="6ec209e9-6c77-4d08-801a-7195b980e41e" containerName="revoke-keys" Nov 03 15:24:55.310742 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.310639 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec209e9-6c77-4d08-801a-7195b980e41e" containerName="revoke-keys" Nov 03 15:24:55.310742 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.310676 4557 cpu_manager.go:414] "containerMap: removing container" podUID="ac5cb6ab-283f-4ff1-a3ea-f8d958451962" containerName="revoke-keys" Nov 03 15:24:55.310742 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.310684 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5cb6ab-283f-4ff1-a3ea-f8d958451962" containerName="revoke-keys" Nov 03 15:24:55.310901 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.310796 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec209e9-6c77-4d08-801a-7195b980e41e" containerName="revoke-keys" Nov 03 15:24:55.310901 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.310813 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5cb6ab-283f-4ff1-a3ea-f8d958451962" containerName="revoke-keys" Nov 03 15:24:55.314322 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.314300 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn" Nov 03 15:24:55.385188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.385108 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn"] Nov 03 15:24:55.418989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.418950 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:24:55.419170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.419009 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:24:55.466440 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.466413 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g899c\" (UniqueName: \"kubernetes.io/projected/a5165cdf-638c-4399-b2db-6526fa123371-kube-api-access-g899c\") pod \"maas-api-revoke-sub-010f6743-kvqmn\" (UID: \"a5165cdf-638c-4399-b2db-6526fa123371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn" Nov 03 15:24:55.466560 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.466486 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5165cdf-638c-4399-b2db-6526fa123371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-010f6743-kvqmn\" (UID: \"a5165cdf-638c-4399-b2db-6526fa123371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn" Nov 03 15:24:55.466626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.466553 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5165cdf-638c-4399-b2db-6526fa123371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-010f6743-kvqmn\" (UID: \"a5165cdf-638c-4399-b2db-6526fa123371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn" Nov 03 15:24:55.567409 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.567376 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5165cdf-638c-4399-b2db-6526fa123371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-010f6743-kvqmn\" (UID: \"a5165cdf-638c-4399-b2db-6526fa123371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn" Nov 03 15:24:55.567409 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.567412 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5165cdf-638c-4399-b2db-6526fa123371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-010f6743-kvqmn\" (UID: \"a5165cdf-638c-4399-b2db-6526fa123371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn" Nov 03 15:24:55.567752 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.567466 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g899c\" (UniqueName: \"kubernetes.io/projected/a5165cdf-638c-4399-b2db-6526fa123371-kube-api-access-g899c\") pod \"maas-api-revoke-sub-010f6743-kvqmn\" (UID: \"a5165cdf-638c-4399-b2db-6526fa123371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn" Nov 03 15:24:55.568164 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.568142 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5165cdf-638c-4399-b2db-6526fa123371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-010f6743-kvqmn\" (UID: \"a5165cdf-638c-4399-b2db-6526fa123371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn" Nov 03 15:24:55.569872 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.569846 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5165cdf-638c-4399-b2db-6526fa123371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-010f6743-kvqmn\" (UID: \"a5165cdf-638c-4399-b2db-6526fa123371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn" Nov 03 15:24:55.631724 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.631689 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g899c\" (UniqueName: \"kubernetes.io/projected/a5165cdf-638c-4399-b2db-6526fa123371-kube-api-access-g899c\") pod \"maas-api-revoke-sub-010f6743-kvqmn\" (UID: \"a5165cdf-638c-4399-b2db-6526fa123371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn" Nov 03 15:24:55.926940 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:55.926907 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn" Nov 03 15:24:56.299881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:56.299856 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn"] Nov 03 15:24:56.301324 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:24:56.301300 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5165cdf_638c_4399_b2db_6526fa123371.slice/crio-66ad8e5064c1385e9788a62d0c2e7463e3b29299e871f9e0258c2fa34be4fa76 WatchSource:0}: Error finding container 66ad8e5064c1385e9788a62d0c2e7463e3b29299e871f9e0258c2fa34be4fa76: Status 404 returned error can't find the container with id 66ad8e5064c1385e9788a62d0c2e7463e3b29299e871f9e0258c2fa34be4fa76 Nov 03 15:24:56.703306 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:56.703274 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn" event={"ID":"a5165cdf-638c-4399-b2db-6526fa123371","Type":"ContainerStarted","Data":"66ad8e5064c1385e9788a62d0c2e7463e3b29299e871f9e0258c2fa34be4fa76"} Nov 03 15:24:57.710706 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:57.710605 4557 generic.go:356] "Generic (PLEG): container finished" podID="a5165cdf-638c-4399-b2db-6526fa123371" containerID="e466ed77882d805189f176fe7e9b1fb8dc2fe987269f057e6c36bc86eeda8b2f" exitCode=0 Nov 03 15:24:57.710706 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:57.710676 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn" event={"ID":"a5165cdf-638c-4399-b2db-6526fa123371","Type":"ContainerDied","Data":"e466ed77882d805189f176fe7e9b1fb8dc2fe987269f057e6c36bc86eeda8b2f"} Nov 03 15:24:58.868353 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:58.868328 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn" Nov 03 15:24:58.976142 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:58.976072 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6"] Nov 03 15:24:58.976519 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:58.976505 4557 cpu_manager.go:414] "containerMap: removing container" podUID="a5165cdf-638c-4399-b2db-6526fa123371" containerName="revoke-keys" Nov 03 15:24:58.976564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:58.976520 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5165cdf-638c-4399-b2db-6526fa123371" containerName="revoke-keys" Nov 03 15:24:58.976625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:58.976616 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5165cdf-638c-4399-b2db-6526fa123371" containerName="revoke-keys" Nov 03 15:24:58.979653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:58.979637 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6" Nov 03 15:24:58.995831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:58.995806 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6"] Nov 03 15:24:59.026791 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.026758 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5165cdf-638c-4399-b2db-6526fa123371-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5165cdf-638c-4399-b2db-6526fa123371-maas-api-xks-ca\") pod \"a5165cdf-638c-4399-b2db-6526fa123371\" (UID: \"a5165cdf-638c-4399-b2db-6526fa123371\") " Nov 03 15:24:59.026942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.026804 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5165cdf-638c-4399-b2db-6526fa123371-kube-api-access-g899c\" (UniqueName: \"kubernetes.io/projected/a5165cdf-638c-4399-b2db-6526fa123371-kube-api-access-g899c\") pod \"a5165cdf-638c-4399-b2db-6526fa123371\" (UID: \"a5165cdf-638c-4399-b2db-6526fa123371\") " Nov 03 15:24:59.026942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.026854 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5165cdf-638c-4399-b2db-6526fa123371-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5165cdf-638c-4399-b2db-6526fa123371-maas-api-openshift-ca\") pod \"a5165cdf-638c-4399-b2db-6526fa123371\" (UID: \"a5165cdf-638c-4399-b2db-6526fa123371\") " Nov 03 15:24:59.027319 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.027293 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5165cdf-638c-4399-b2db-6526fa123371-maas-api-openshift-ca" pod "a5165cdf-638c-4399-b2db-6526fa123371" (UID: "a5165cdf-638c-4399-b2db-6526fa123371"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:24:59.028888 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.028861 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5165cdf-638c-4399-b2db-6526fa123371-maas-api-xks-ca" pod "a5165cdf-638c-4399-b2db-6526fa123371" (UID: "a5165cdf-638c-4399-b2db-6526fa123371"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:24:59.028991 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.028960 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5165cdf-638c-4399-b2db-6526fa123371-kube-api-access-g899c" pod "a5165cdf-638c-4399-b2db-6526fa123371" (UID: "a5165cdf-638c-4399-b2db-6526fa123371"). InnerVolumeSpecName "kube-api-access-g899c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:24:59.127658 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.127622 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49de6b74-9l6d6\" (UID: \"f1b715d1-2833-48d1-a1fc-e30c1abcf0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6" Nov 03 15:24:59.127827 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.127674 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjvxc\" (UniqueName: \"kubernetes.io/projected/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-kube-api-access-rjvxc\") pod \"maas-api-revoke-sub-49de6b74-9l6d6\" (UID: \"f1b715d1-2833-48d1-a1fc-e30c1abcf0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6" Nov 03 15:24:59.127875 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.127825 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49de6b74-9l6d6\" (UID: \"f1b715d1-2833-48d1-a1fc-e30c1abcf0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6" Nov 03 15:24:59.127939 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.127925 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5165cdf-638c-4399-b2db-6526fa123371-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:59.127980 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.127945 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5165cdf-638c-4399-b2db-6526fa123371-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:59.127980 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.127955 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g899c\" (UniqueName: \"kubernetes.io/projected/a5165cdf-638c-4399-b2db-6526fa123371-kube-api-access-g899c\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:24:59.229341 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.229247 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49de6b74-9l6d6\" (UID: \"f1b715d1-2833-48d1-a1fc-e30c1abcf0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6" Nov 03 15:24:59.229341 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.229331 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49de6b74-9l6d6\" (UID: \"f1b715d1-2833-48d1-a1fc-e30c1abcf0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6" Nov 03 15:24:59.229566 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.229377 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjvxc\" (UniqueName: \"kubernetes.io/projected/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-kube-api-access-rjvxc\") pod \"maas-api-revoke-sub-49de6b74-9l6d6\" (UID: \"f1b715d1-2833-48d1-a1fc-e30c1abcf0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6" Nov 03 15:24:59.230095 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.230074 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49de6b74-9l6d6\" (UID: \"f1b715d1-2833-48d1-a1fc-e30c1abcf0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6" Nov 03 15:24:59.231655 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.231638 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49de6b74-9l6d6\" (UID: \"f1b715d1-2833-48d1-a1fc-e30c1abcf0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6" Nov 03 15:24:59.290496 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.290469 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjvxc\" (UniqueName: \"kubernetes.io/projected/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-kube-api-access-rjvxc\") pod \"maas-api-revoke-sub-49de6b74-9l6d6\" (UID: \"f1b715d1-2833-48d1-a1fc-e30c1abcf0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6" Nov 03 15:24:59.293270 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.293253 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6" Nov 03 15:24:59.668097 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.668072 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6"] Nov 03 15:24:59.670111 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:24:59.670063 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b715d1_2833_48d1_a1fc_e30c1abcf0ab.slice/crio-ebaecaf41b065619cd2862f8a24341d929672e3046130a68c217c59e59078361 WatchSource:0}: Error finding container ebaecaf41b065619cd2862f8a24341d929672e3046130a68c217c59e59078361: Status 404 returned error can't find the container with id ebaecaf41b065619cd2862f8a24341d929672e3046130a68c217c59e59078361 Nov 03 15:24:59.722827 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.722797 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6" event={"ID":"f1b715d1-2833-48d1-a1fc-e30c1abcf0ab","Type":"ContainerStarted","Data":"ebaecaf41b065619cd2862f8a24341d929672e3046130a68c217c59e59078361"} Nov 03 15:24:59.724111 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.724093 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn" Nov 03 15:24:59.724201 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.724125 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn" event={"ID":"a5165cdf-638c-4399-b2db-6526fa123371","Type":"ContainerDied","Data":"66ad8e5064c1385e9788a62d0c2e7463e3b29299e871f9e0258c2fa34be4fa76"} Nov 03 15:24:59.724201 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:24:59.724160 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66ad8e5064c1385e9788a62d0c2e7463e3b29299e871f9e0258c2fa34be4fa76" Nov 03 15:25:00.057654 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:00.054798 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:25:00.730910 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:00.730873 4557 generic.go:356] "Generic (PLEG): container finished" podID="f1b715d1-2833-48d1-a1fc-e30c1abcf0ab" containerID="e371108c04645515d762de1c13024ce4bfe2c7770a112653f8eba6704a674a64" exitCode=0 Nov 03 15:25:00.731077 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:00.730918 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6" event={"ID":"f1b715d1-2833-48d1-a1fc-e30c1abcf0ab","Type":"ContainerDied","Data":"e371108c04645515d762de1c13024ce4bfe2c7770a112653f8eba6704a674a64"} Nov 03 15:25:00.860333 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:00.860250 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:25:01.643355 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.643315 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5"] Nov 03 15:25:01.647127 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.647106 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5" Nov 03 15:25:01.751035 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.751007 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5"] Nov 03 15:25:01.754638 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.754607 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5b7c154-26fg5\" (UID: \"43c8ee92-ba14-4954-a3fd-fc2e492b8a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5" Nov 03 15:25:01.754781 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.754759 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5b7c154-26fg5\" (UID: \"43c8ee92-ba14-4954-a3fd-fc2e492b8a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5" Nov 03 15:25:01.754845 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.754806 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js6w8\" (UniqueName: \"kubernetes.io/projected/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-kube-api-access-js6w8\") pod \"maas-api-revoke-sub-a5b7c154-26fg5\" (UID: \"43c8ee92-ba14-4954-a3fd-fc2e492b8a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5" Nov 03 15:25:01.855636 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.855607 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5b7c154-26fg5\" (UID: \"43c8ee92-ba14-4954-a3fd-fc2e492b8a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5" Nov 03 15:25:01.855788 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.855670 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5b7c154-26fg5\" (UID: \"43c8ee92-ba14-4954-a3fd-fc2e492b8a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5" Nov 03 15:25:01.855788 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.855694 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js6w8\" (UniqueName: \"kubernetes.io/projected/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-kube-api-access-js6w8\") pod \"maas-api-revoke-sub-a5b7c154-26fg5\" (UID: \"43c8ee92-ba14-4954-a3fd-fc2e492b8a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5" Nov 03 15:25:01.856295 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.856270 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5b7c154-26fg5\" (UID: \"43c8ee92-ba14-4954-a3fd-fc2e492b8a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5" Nov 03 15:25:01.858056 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.858034 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5b7c154-26fg5\" (UID: \"43c8ee92-ba14-4954-a3fd-fc2e492b8a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5" Nov 03 15:25:01.876534 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.876507 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6" Nov 03 15:25:01.917459 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.917389 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js6w8\" (UniqueName: \"kubernetes.io/projected/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-kube-api-access-js6w8\") pod \"maas-api-revoke-sub-a5b7c154-26fg5\" (UID: \"43c8ee92-ba14-4954-a3fd-fc2e492b8a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5" Nov 03 15:25:01.956960 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.956927 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-kube-api-access-rjvxc\" (UniqueName: \"kubernetes.io/projected/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-kube-api-access-rjvxc\") pod \"f1b715d1-2833-48d1-a1fc-e30c1abcf0ab\" (UID: \"f1b715d1-2833-48d1-a1fc-e30c1abcf0ab\") " Nov 03 15:25:01.957105 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.957082 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-maas-api-xks-ca\") pod \"f1b715d1-2833-48d1-a1fc-e30c1abcf0ab\" (UID: \"f1b715d1-2833-48d1-a1fc-e30c1abcf0ab\") " Nov 03 15:25:01.957216 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.957200 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-maas-api-openshift-ca\") pod \"f1b715d1-2833-48d1-a1fc-e30c1abcf0ab\" (UID: \"f1b715d1-2833-48d1-a1fc-e30c1abcf0ab\") " Nov 03 15:25:01.957610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.957566 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-maas-api-openshift-ca" pod "f1b715d1-2833-48d1-a1fc-e30c1abcf0ab" (UID: "f1b715d1-2833-48d1-a1fc-e30c1abcf0ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:25:01.959058 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.959035 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-maas-api-xks-ca" pod "f1b715d1-2833-48d1-a1fc-e30c1abcf0ab" (UID: "f1b715d1-2833-48d1-a1fc-e30c1abcf0ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:25:01.959146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.959037 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-kube-api-access-rjvxc" pod "f1b715d1-2833-48d1-a1fc-e30c1abcf0ab" (UID: "f1b715d1-2833-48d1-a1fc-e30c1abcf0ab"). InnerVolumeSpecName "kube-api-access-rjvxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:25:01.960148 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:01.960123 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5" Nov 03 15:25:02.058848 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:02.058812 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:02.058848 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:02.058847 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjvxc\" (UniqueName: \"kubernetes.io/projected/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-kube-api-access-rjvxc\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:02.058848 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:02.058857 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:02.343526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:02.343494 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5"] Nov 03 15:25:02.344659 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:25:02.344633 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c8ee92_ba14_4954_a3fd_fc2e492b8a05.slice/crio-8723865166b2fb232c5dea89850e1e9284d92d7d49cc1dc5af39212516363e58 WatchSource:0}: Error finding container 8723865166b2fb232c5dea89850e1e9284d92d7d49cc1dc5af39212516363e58: Status 404 returned error can't find the container with id 8723865166b2fb232c5dea89850e1e9284d92d7d49cc1dc5af39212516363e58 Nov 03 15:25:02.744516 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:02.744477 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5" event={"ID":"43c8ee92-ba14-4954-a3fd-fc2e492b8a05","Type":"ContainerStarted","Data":"1d7c61069d1575661344ae24f53c32c6c1cb7a8bc85b9ae848a5f2cd337b2e33"} Nov 03 15:25:02.744984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:02.744570 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5" event={"ID":"43c8ee92-ba14-4954-a3fd-fc2e492b8a05","Type":"ContainerStarted","Data":"8723865166b2fb232c5dea89850e1e9284d92d7d49cc1dc5af39212516363e58"} Nov 03 15:25:02.745851 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:02.745831 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6" Nov 03 15:25:02.745920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:02.745857 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6" event={"ID":"f1b715d1-2833-48d1-a1fc-e30c1abcf0ab","Type":"ContainerDied","Data":"ebaecaf41b065619cd2862f8a24341d929672e3046130a68c217c59e59078361"} Nov 03 15:25:02.745920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:02.745894 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebaecaf41b065619cd2862f8a24341d929672e3046130a68c217c59e59078361" Nov 03 15:25:02.862531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:02.862431 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5" podStartSLOduration=1.8624169990000001 podStartE2EDuration="1.862416999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:02.861062784 +0000 UTC m=+1012.704184323" watchObservedRunningTime="2026-11-03 15:25:02.862416999 +0000 UTC m=+1012.705538542" Nov 03 15:25:03.753099 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:03.753069 4557 generic.go:356] "Generic (PLEG): container finished" podID="43c8ee92-ba14-4954-a3fd-fc2e492b8a05" containerID="1d7c61069d1575661344ae24f53c32c6c1cb7a8bc85b9ae848a5f2cd337b2e33" exitCode=0 Nov 03 15:25:03.753497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:03.753163 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5" event={"ID":"43c8ee92-ba14-4954-a3fd-fc2e492b8a05","Type":"ContainerDied","Data":"1d7c61069d1575661344ae24f53c32c6c1cb7a8bc85b9ae848a5f2cd337b2e33"} Nov 03 15:25:04.856036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:04.856005 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:25:04.925941 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:04.925915 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5" Nov 03 15:25:05.088937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:05.088902 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-maas-api-openshift-ca\") pod \"43c8ee92-ba14-4954-a3fd-fc2e492b8a05\" (UID: \"43c8ee92-ba14-4954-a3fd-fc2e492b8a05\") " Nov 03 15:25:05.089128 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:05.089022 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-maas-api-xks-ca\") pod \"43c8ee92-ba14-4954-a3fd-fc2e492b8a05\" (UID: \"43c8ee92-ba14-4954-a3fd-fc2e492b8a05\") " Nov 03 15:25:05.089128 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:05.089072 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-kube-api-access-js6w8\" (UniqueName: \"kubernetes.io/projected/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-kube-api-access-js6w8\") pod \"43c8ee92-ba14-4954-a3fd-fc2e492b8a05\" (UID: \"43c8ee92-ba14-4954-a3fd-fc2e492b8a05\") " Nov 03 15:25:05.089280 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:05.089255 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-maas-api-openshift-ca" pod "43c8ee92-ba14-4954-a3fd-fc2e492b8a05" (UID: "43c8ee92-ba14-4954-a3fd-fc2e492b8a05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:25:05.089471 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:05.089448 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:05.091035 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:05.091011 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-maas-api-xks-ca" pod "43c8ee92-ba14-4954-a3fd-fc2e492b8a05" (UID: "43c8ee92-ba14-4954-a3fd-fc2e492b8a05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:25:05.091146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:05.091123 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-kube-api-access-js6w8" pod "43c8ee92-ba14-4954-a3fd-fc2e492b8a05" (UID: "43c8ee92-ba14-4954-a3fd-fc2e492b8a05"). InnerVolumeSpecName "kube-api-access-js6w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:25:05.190510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:05.190444 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:05.190510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:05.190480 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js6w8\" (UniqueName: \"kubernetes.io/projected/43c8ee92-ba14-4954-a3fd-fc2e492b8a05-kube-api-access-js6w8\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:05.765626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:05.765576 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5" Nov 03 15:25:05.765835 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:05.765622 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5" event={"ID":"43c8ee92-ba14-4954-a3fd-fc2e492b8a05","Type":"ContainerDied","Data":"8723865166b2fb232c5dea89850e1e9284d92d7d49cc1dc5af39212516363e58"} Nov 03 15:25:05.765835 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:05.765670 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8723865166b2fb232c5dea89850e1e9284d92d7d49cc1dc5af39212516363e58" Nov 03 15:25:09.711276 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.711248 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9"] Nov 03 15:25:09.711695 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.711652 4557 cpu_manager.go:414] "containerMap: removing container" podUID="43c8ee92-ba14-4954-a3fd-fc2e492b8a05" containerName="revoke-keys" Nov 03 15:25:09.711695 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.711664 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c8ee92-ba14-4954-a3fd-fc2e492b8a05" containerName="revoke-keys" Nov 03 15:25:09.711695 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.711693 4557 cpu_manager.go:414] "containerMap: removing container" podUID="f1b715d1-2833-48d1-a1fc-e30c1abcf0ab" containerName="revoke-keys" Nov 03 15:25:09.711802 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.711697 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b715d1-2833-48d1-a1fc-e30c1abcf0ab" containerName="revoke-keys" Nov 03 15:25:09.711802 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.711756 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c8ee92-ba14-4954-a3fd-fc2e492b8a05" containerName="revoke-keys" Nov 03 15:25:09.711802 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.711767 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b715d1-2833-48d1-a1fc-e30c1abcf0ab" containerName="revoke-keys" Nov 03 15:25:09.714944 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.714925 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9" Nov 03 15:25:09.733864 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.733839 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ceb986-47b2-4326-ac8b-a664a425c81c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93ac7341-bczk9\" (UID: \"c5ceb986-47b2-4326-ac8b-a664a425c81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9" Nov 03 15:25:09.733941 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.733876 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv2vp\" (UniqueName: \"kubernetes.io/projected/c5ceb986-47b2-4326-ac8b-a664a425c81c-kube-api-access-jv2vp\") pod \"maas-api-revoke-sub-93ac7341-bczk9\" (UID: \"c5ceb986-47b2-4326-ac8b-a664a425c81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9" Nov 03 15:25:09.733941 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.733906 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ceb986-47b2-4326-ac8b-a664a425c81c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93ac7341-bczk9\" (UID: \"c5ceb986-47b2-4326-ac8b-a664a425c81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9" Nov 03 15:25:09.770061 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.770014 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:25:09.773837 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.773813 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9"] Nov 03 15:25:09.810320 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.810278 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:25:09.819793 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.819757 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:25:09.834471 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.834445 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv2vp\" (UniqueName: \"kubernetes.io/projected/c5ceb986-47b2-4326-ac8b-a664a425c81c-kube-api-access-jv2vp\") pod \"maas-api-revoke-sub-93ac7341-bczk9\" (UID: \"c5ceb986-47b2-4326-ac8b-a664a425c81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9" Nov 03 15:25:09.834620 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.834489 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ceb986-47b2-4326-ac8b-a664a425c81c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93ac7341-bczk9\" (UID: \"c5ceb986-47b2-4326-ac8b-a664a425c81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9" Nov 03 15:25:09.834708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.834663 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ceb986-47b2-4326-ac8b-a664a425c81c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93ac7341-bczk9\" (UID: \"c5ceb986-47b2-4326-ac8b-a664a425c81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9" Nov 03 15:25:09.835288 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.835265 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ceb986-47b2-4326-ac8b-a664a425c81c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93ac7341-bczk9\" (UID: \"c5ceb986-47b2-4326-ac8b-a664a425c81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9" Nov 03 15:25:09.836762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.836742 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ceb986-47b2-4326-ac8b-a664a425c81c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93ac7341-bczk9\" (UID: \"c5ceb986-47b2-4326-ac8b-a664a425c81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9" Nov 03 15:25:09.895884 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:09.895862 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv2vp\" (UniqueName: \"kubernetes.io/projected/c5ceb986-47b2-4326-ac8b-a664a425c81c-kube-api-access-jv2vp\") pod \"maas-api-revoke-sub-93ac7341-bczk9\" (UID: \"c5ceb986-47b2-4326-ac8b-a664a425c81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9" Nov 03 15:25:10.027920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:10.027828 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9" Nov 03 15:25:10.404105 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:10.404078 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9"] Nov 03 15:25:10.405344 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:25:10.405322 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ceb986_47b2_4326_ac8b_a664a425c81c.slice/crio-0804bcb9fc109b20b33305e418cf3924b5978896c30d042fc7ebca2bc48d0689 WatchSource:0}: Error finding container 0804bcb9fc109b20b33305e418cf3924b5978896c30d042fc7ebca2bc48d0689: Status 404 returned error can't find the container with id 0804bcb9fc109b20b33305e418cf3924b5978896c30d042fc7ebca2bc48d0689 Nov 03 15:25:10.793771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:10.793736 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9" event={"ID":"c5ceb986-47b2-4326-ac8b-a664a425c81c","Type":"ContainerStarted","Data":"cc84cebd8dfb4065b7f1c3d83ea97b5c3a34934a377706c11d2de2b94d33b493"} Nov 03 15:25:10.794153 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:10.793795 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9" event={"ID":"c5ceb986-47b2-4326-ac8b-a664a425c81c","Type":"ContainerStarted","Data":"0804bcb9fc109b20b33305e418cf3924b5978896c30d042fc7ebca2bc48d0689"} Nov 03 15:25:10.911163 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:10.911111 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9" podStartSLOduration=1.911093346 podStartE2EDuration="1.911093346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:10.909957191 +0000 UTC m=+1020.753078733" watchObservedRunningTime="2026-11-03 15:25:10.911093346 +0000 UTC m=+1020.754214889" Nov 03 15:25:11.664423 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.664326 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4"] Nov 03 15:25:11.668163 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.668142 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.750761 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.750729 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.750761 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.750766 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.750963 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.750876 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.750963 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.750945 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.751033 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.750965 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.751033 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.750992 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.751133 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.751080 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh8x9\" (UniqueName: \"kubernetes.io/projected/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-kube-api-access-rh8x9\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.751133 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.751114 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.751227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.751144 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.783656 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.783626 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4"] Nov 03 15:25:11.801328 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.801299 4557 generic.go:356] "Generic (PLEG): container finished" podID="c5ceb986-47b2-4326-ac8b-a664a425c81c" containerID="cc84cebd8dfb4065b7f1c3d83ea97b5c3a34934a377706c11d2de2b94d33b493" exitCode=0 Nov 03 15:25:11.801703 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.801347 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9" event={"ID":"c5ceb986-47b2-4326-ac8b-a664a425c81c","Type":"ContainerDied","Data":"cc84cebd8dfb4065b7f1c3d83ea97b5c3a34934a377706c11d2de2b94d33b493"} Nov 03 15:25:11.852365 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.852331 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.852532 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.852374 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.852532 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.852435 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.852532 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.852474 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.852532 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.852496 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.852786 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.852534 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.852786 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.852610 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh8x9\" (UniqueName: \"kubernetes.io/projected/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-kube-api-access-rh8x9\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.852786 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.852661 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.852786 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.852705 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.853039 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.853014 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.853120 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.853091 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.853180 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.853136 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.853180 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.853153 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.853346 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.853328 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.854723 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.854700 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.855005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.854989 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.917560 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.917491 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh8x9\" (UniqueName: \"kubernetes.io/projected/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-kube-api-access-rh8x9\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.917738 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.917599 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-zgkl4\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:11.981225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:11.981194 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:12.410405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.410374 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4"] Nov 03 15:25:12.411630 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:25:12.411604 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72286ed3_dafb_4c80_b6bf_6db70a73f2a6.slice/crio-fe7737f9095c6c20bc5b210f665fbeb93b2fcbae999082202eab0ba1c3c1c99d WatchSource:0}: Error finding container fe7737f9095c6c20bc5b210f665fbeb93b2fcbae999082202eab0ba1c3c1c99d: Status 404 returned error can't find the container with id fe7737f9095c6c20bc5b210f665fbeb93b2fcbae999082202eab0ba1c3c1c99d Nov 03 15:25:12.414014 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.413986 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:25:12.414085 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.414050 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:25:12.414085 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.414077 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:25:12.479977 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.479952 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8"] Nov 03 15:25:12.483788 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.483767 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8" Nov 03 15:25:12.553387 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.553351 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8"] Nov 03 15:25:12.557360 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.557325 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7b378ea-pllv8\" (UID: \"382b7b36-beb8-4b8d-aa48-4071c3cd4f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8" Nov 03 15:25:12.557608 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.557571 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkzl7\" (UniqueName: \"kubernetes.io/projected/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-kube-api-access-kkzl7\") pod \"maas-api-revoke-sub-e7b378ea-pllv8\" (UID: \"382b7b36-beb8-4b8d-aa48-4071c3cd4f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8" Nov 03 15:25:12.557778 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.557757 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7b378ea-pllv8\" (UID: \"382b7b36-beb8-4b8d-aa48-4071c3cd4f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8" Nov 03 15:25:12.658445 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.658410 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7b378ea-pllv8\" (UID: \"382b7b36-beb8-4b8d-aa48-4071c3cd4f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8" Nov 03 15:25:12.658638 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.658454 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkzl7\" (UniqueName: \"kubernetes.io/projected/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-kube-api-access-kkzl7\") pod \"maas-api-revoke-sub-e7b378ea-pllv8\" (UID: \"382b7b36-beb8-4b8d-aa48-4071c3cd4f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8" Nov 03 15:25:12.658638 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.658495 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7b378ea-pllv8\" (UID: \"382b7b36-beb8-4b8d-aa48-4071c3cd4f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8" Nov 03 15:25:12.659275 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.659217 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7b378ea-pllv8\" (UID: \"382b7b36-beb8-4b8d-aa48-4071c3cd4f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8" Nov 03 15:25:12.660696 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.660677 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7b378ea-pllv8\" (UID: \"382b7b36-beb8-4b8d-aa48-4071c3cd4f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8" Nov 03 15:25:12.720458 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.720424 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkzl7\" (UniqueName: \"kubernetes.io/projected/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-kube-api-access-kkzl7\") pod \"maas-api-revoke-sub-e7b378ea-pllv8\" (UID: \"382b7b36-beb8-4b8d-aa48-4071c3cd4f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8" Nov 03 15:25:12.800419 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.800383 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8" Nov 03 15:25:12.808869 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.808832 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" event={"ID":"72286ed3-dafb-4c80-b6bf-6db70a73f2a6","Type":"ContainerStarted","Data":"6a6c373a0fe1db401066e573fb433fa979eb8ae7eaaa64440cc37272d45face5"} Nov 03 15:25:12.809314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.808907 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" event={"ID":"72286ed3-dafb-4c80-b6bf-6db70a73f2a6","Type":"ContainerStarted","Data":"fe7737f9095c6c20bc5b210f665fbeb93b2fcbae999082202eab0ba1c3c1c99d"} Nov 03 15:25:12.936381 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.936209 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" podStartSLOduration=1.936193176 podStartE2EDuration="1.936193176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:12.935187567 +0000 UTC m=+1022.778309109" watchObservedRunningTime="2026-11-03 15:25:12.936193176 +0000 UTC m=+1022.779314719" Nov 03 15:25:12.985786 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:12.983960 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:13.009128 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:13.009098 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9" Nov 03 15:25:13.065241 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:13.065212 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ceb986-47b2-4326-ac8b-a664a425c81c-kube-api-access-jv2vp\" (UniqueName: \"kubernetes.io/projected/c5ceb986-47b2-4326-ac8b-a664a425c81c-kube-api-access-jv2vp\") pod \"c5ceb986-47b2-4326-ac8b-a664a425c81c\" (UID: \"c5ceb986-47b2-4326-ac8b-a664a425c81c\") " Nov 03 15:25:13.065504 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:13.065276 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ceb986-47b2-4326-ac8b-a664a425c81c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ceb986-47b2-4326-ac8b-a664a425c81c-maas-api-openshift-ca\") pod \"c5ceb986-47b2-4326-ac8b-a664a425c81c\" (UID: \"c5ceb986-47b2-4326-ac8b-a664a425c81c\") " Nov 03 15:25:13.065504 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:13.065376 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5ceb986-47b2-4326-ac8b-a664a425c81c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ceb986-47b2-4326-ac8b-a664a425c81c-maas-api-xks-ca\") pod \"c5ceb986-47b2-4326-ac8b-a664a425c81c\" (UID: \"c5ceb986-47b2-4326-ac8b-a664a425c81c\") " Nov 03 15:25:13.065713 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:13.065677 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ceb986-47b2-4326-ac8b-a664a425c81c-maas-api-openshift-ca" pod "c5ceb986-47b2-4326-ac8b-a664a425c81c" (UID: "c5ceb986-47b2-4326-ac8b-a664a425c81c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:25:13.065851 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:13.065835 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ceb986-47b2-4326-ac8b-a664a425c81c-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:13.067360 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:13.067335 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ceb986-47b2-4326-ac8b-a664a425c81c-kube-api-access-jv2vp" pod "c5ceb986-47b2-4326-ac8b-a664a425c81c" (UID: "c5ceb986-47b2-4326-ac8b-a664a425c81c"). InnerVolumeSpecName "kube-api-access-jv2vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:25:13.067440 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:13.067355 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ceb986-47b2-4326-ac8b-a664a425c81c-maas-api-xks-ca" pod "c5ceb986-47b2-4326-ac8b-a664a425c81c" (UID: "c5ceb986-47b2-4326-ac8b-a664a425c81c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:25:13.166530 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:13.166496 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv2vp\" (UniqueName: \"kubernetes.io/projected/c5ceb986-47b2-4326-ac8b-a664a425c81c-kube-api-access-jv2vp\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:13.166530 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:13.166528 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ceb986-47b2-4326-ac8b-a664a425c81c-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:13.205953 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:13.205919 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8"] Nov 03 15:25:13.815707 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:13.815675 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9" Nov 03 15:25:13.816133 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:13.815679 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9" event={"ID":"c5ceb986-47b2-4326-ac8b-a664a425c81c","Type":"ContainerDied","Data":"0804bcb9fc109b20b33305e418cf3924b5978896c30d042fc7ebca2bc48d0689"} Nov 03 15:25:13.816133 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:13.815852 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0804bcb9fc109b20b33305e418cf3924b5978896c30d042fc7ebca2bc48d0689" Nov 03 15:25:13.817325 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:13.817293 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8" event={"ID":"382b7b36-beb8-4b8d-aa48-4071c3cd4f10","Type":"ContainerStarted","Data":"ee877b497291ac84b1ba86e77fe8d9781766a7f9fc0f4f41dd5b29b41ca97855"} Nov 03 15:25:13.817429 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:13.817350 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8" event={"ID":"382b7b36-beb8-4b8d-aa48-4071c3cd4f10","Type":"ContainerStarted","Data":"35eca69202ae1b7e81300bfa409489be7f144ba5fb2f3d8d3273ac51dad6a006"} Nov 03 15:25:13.936421 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:13.936325 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8" podStartSLOduration=1.936313519 podStartE2EDuration="1.936313519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:13.935181723 +0000 UTC m=+1023.778303265" watchObservedRunningTime="2026-11-03 15:25:13.936313519 +0000 UTC m=+1023.779435057" Nov 03 15:25:14.002664 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:14.002627 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-zgkl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 15:25:14.002839 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:14.002698 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" podUID="72286ed3-dafb-4c80-b6bf-6db70a73f2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 15:25:14.716512 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:14.716481 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:25:14.825317 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:14.825286 4557 generic.go:356] "Generic (PLEG): container finished" podID="382b7b36-beb8-4b8d-aa48-4071c3cd4f10" containerID="ee877b497291ac84b1ba86e77fe8d9781766a7f9fc0f4f41dd5b29b41ca97855" exitCode=0 Nov 03 15:25:14.825681 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:14.825381 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8" event={"ID":"382b7b36-beb8-4b8d-aa48-4071c3cd4f10","Type":"ContainerDied","Data":"ee877b497291ac84b1ba86e77fe8d9781766a7f9fc0f4f41dd5b29b41ca97855"} Nov 03 15:25:14.989658 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:14.987633 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-zgkl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 15:25:14.989658 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:14.987719 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" podUID="72286ed3-dafb-4c80-b6bf-6db70a73f2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 15:25:15.568235 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.568202 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8"] Nov 03 15:25:15.568669 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.568654 4557 cpu_manager.go:414] "containerMap: removing container" podUID="c5ceb986-47b2-4326-ac8b-a664a425c81c" containerName="revoke-keys" Nov 03 15:25:15.568721 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.568673 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ceb986-47b2-4326-ac8b-a664a425c81c" containerName="revoke-keys" Nov 03 15:25:15.568769 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.568760 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ceb986-47b2-4326-ac8b-a664a425c81c" containerName="revoke-keys" Nov 03 15:25:15.572084 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.572068 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.611608 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.611553 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.611753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.611683 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.611753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.611736 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl8dn\" (UniqueName: \"kubernetes.io/projected/7620cddf-1590-4162-a61a-71fad3ef52ae-kube-api-access-pl8dn\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.611823 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.611760 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.611866 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.611817 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.611899 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.611870 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7620cddf-1590-4162-a61a-71fad3ef52ae-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.611950 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.611938 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.611994 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.611971 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.612025 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.612004 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.674669 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.674635 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-f59d26-openshift-default-dockercfg-nkdw5\"" Nov 03 15:25:15.692606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.678412 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8"] Nov 03 15:25:15.713072 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.713041 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.713239 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.713092 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.713239 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.713127 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl8dn\" (UniqueName: \"kubernetes.io/projected/7620cddf-1590-4162-a61a-71fad3ef52ae-kube-api-access-pl8dn\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.713239 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.713158 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.713239 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.713177 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.713239 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.713212 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7620cddf-1590-4162-a61a-71fad3ef52ae-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.713508 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.713251 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.713508 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.713289 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.713639 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.713570 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.713704 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.713635 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.713761 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.713708 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.713887 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.713868 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.714067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.714040 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.714179 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.714161 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7620cddf-1590-4162-a61a-71fad3ef52ae-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.715692 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.715670 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.715922 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.715904 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.777817 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.777780 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl8dn\" (UniqueName: \"kubernetes.io/projected/7620cddf-1590-4162-a61a-71fad3ef52ae-kube-api-access-pl8dn\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.780050 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.780020 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:15.909915 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:15.909888 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:16.006760 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.006717 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-zgkl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 15:25:16.006900 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.006778 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" podUID="72286ed3-dafb-4c80-b6bf-6db70a73f2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 15:25:16.022624 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.022318 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8" Nov 03 15:25:16.119326 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.119228 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-maas-api-openshift-ca\") pod \"382b7b36-beb8-4b8d-aa48-4071c3cd4f10\" (UID: \"382b7b36-beb8-4b8d-aa48-4071c3cd4f10\") " Nov 03 15:25:16.119497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.119426 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-kube-api-access-kkzl7\" (UniqueName: \"kubernetes.io/projected/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-kube-api-access-kkzl7\") pod \"382b7b36-beb8-4b8d-aa48-4071c3cd4f10\" (UID: \"382b7b36-beb8-4b8d-aa48-4071c3cd4f10\") " Nov 03 15:25:16.119497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.119459 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-maas-api-xks-ca\") pod \"382b7b36-beb8-4b8d-aa48-4071c3cd4f10\" (UID: \"382b7b36-beb8-4b8d-aa48-4071c3cd4f10\") " Nov 03 15:25:16.119644 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.119534 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-maas-api-openshift-ca" pod "382b7b36-beb8-4b8d-aa48-4071c3cd4f10" (UID: "382b7b36-beb8-4b8d-aa48-4071c3cd4f10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:25:16.119828 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.119809 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:16.121438 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.121414 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-kube-api-access-kkzl7" pod "382b7b36-beb8-4b8d-aa48-4071c3cd4f10" (UID: "382b7b36-beb8-4b8d-aa48-4071c3cd4f10"). InnerVolumeSpecName "kube-api-access-kkzl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:25:16.121438 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.121431 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-maas-api-xks-ca" pod "382b7b36-beb8-4b8d-aa48-4071c3cd4f10" (UID: "382b7b36-beb8-4b8d-aa48-4071c3cd4f10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:25:16.220734 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.220701 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkzl7\" (UniqueName: \"kubernetes.io/projected/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-kube-api-access-kkzl7\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:16.220734 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.220728 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382b7b36-beb8-4b8d-aa48-4071c3cd4f10-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:16.361846 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.361815 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8"] Nov 03 15:25:16.362940 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:25:16.362916 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7620cddf_1590_4162_a61a_71fad3ef52ae.slice/crio-36c3de1383e316c7398c1e667415ab2cbef46ff21ef40ef4920216d31c2e7286 WatchSource:0}: Error finding container 36c3de1383e316c7398c1e667415ab2cbef46ff21ef40ef4920216d31c2e7286: Status 404 returned error can't find the container with id 36c3de1383e316c7398c1e667415ab2cbef46ff21ef40ef4920216d31c2e7286 Nov 03 15:25:16.365303 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.365275 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:25:16.365383 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.365333 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:25:16.365383 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.365361 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:25:16.539886 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:25:16.539850 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:25:16.839416 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.839377 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" event={"ID":"7620cddf-1590-4162-a61a-71fad3ef52ae","Type":"ContainerStarted","Data":"eca0b29d091730481266ab847eab93d1459da0c8e339131bdf23761ee5afa4be"} Nov 03 15:25:16.839574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.839458 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" event={"ID":"7620cddf-1590-4162-a61a-71fad3ef52ae","Type":"ContainerStarted","Data":"36c3de1383e316c7398c1e667415ab2cbef46ff21ef40ef4920216d31c2e7286"} Nov 03 15:25:16.840766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.840741 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8" event={"ID":"382b7b36-beb8-4b8d-aa48-4071c3cd4f10","Type":"ContainerDied","Data":"35eca69202ae1b7e81300bfa409489be7f144ba5fb2f3d8d3273ac51dad6a006"} Nov 03 15:25:16.840823 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.840789 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8" Nov 03 15:25:16.840883 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.840789 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35eca69202ae1b7e81300bfa409489be7f144ba5fb2f3d8d3273ac51dad6a006" Nov 03 15:25:16.910970 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.910938 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:16.963702 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.963655 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" podStartSLOduration=1.963638021 podStartE2EDuration="1.963638021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:16.960694315 +0000 UTC m=+1026.803815857" watchObservedRunningTime="2026-11-03 15:25:16.963638021 +0000 UTC m=+1026.806759614" Nov 03 15:25:16.996657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.993967 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-zgkl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 15:25:16.996657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:16.994032 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" podUID="72286ed3-dafb-4c80-b6bf-6db70a73f2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 15:25:17.919268 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:17.914691 4557 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 15:25:17.919268 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:17.914753 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" podUID="7620cddf-1590-4162-a61a-71fad3ef52ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 15:25:17.995606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:17.993681 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-zgkl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 15:25:17.995606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:17.993740 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" podUID="72286ed3-dafb-4c80-b6bf-6db70a73f2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 15:25:18.911608 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:18.911329 4557 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 15:25:18.911608 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:18.911424 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" podUID="7620cddf-1590-4162-a61a-71fad3ef52ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 15:25:18.987604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:18.983954 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-zgkl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 15:25:18.987604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:18.984048 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" podUID="72286ed3-dafb-4c80-b6bf-6db70a73f2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 15:25:19.767612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:19.764953 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:25:19.912605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:19.911973 4557 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 15:25:19.912605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:19.912046 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" podUID="7620cddf-1590-4162-a61a-71fad3ef52ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 15:25:19.987605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:19.982919 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-zgkl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 15:25:19.987605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:19.983002 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" podUID="72286ed3-dafb-4c80-b6bf-6db70a73f2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 15:25:20.911275 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:20.911241 4557 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 15:25:20.911461 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:20.911302 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" podUID="7620cddf-1590-4162-a61a-71fad3ef52ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 15:25:21.303620 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:21.302967 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:21.305056 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:21.304914 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:21.309362 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:21.308777 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:25:21.910708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:21.910674 4557 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 15:25:21.910866 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:21.910733 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" podUID="7620cddf-1590-4162-a61a-71fad3ef52ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 15:25:22.910318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:22.910285 4557 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 15:25:22.910762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:22.910335 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" podUID="7620cddf-1590-4162-a61a-71fad3ef52ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 15:25:24.069742 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:24.069713 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:24.070155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:24.070027 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:24.071030 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:24.070917 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:25:24.736997 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:24.736961 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:25:26.444993 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:26.444957 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc"] Nov 03 15:25:26.445443 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:26.445411 4557 cpu_manager.go:414] "containerMap: removing container" podUID="382b7b36-beb8-4b8d-aa48-4071c3cd4f10" containerName="revoke-keys" Nov 03 15:25:26.445443 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:26.445422 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="382b7b36-beb8-4b8d-aa48-4071c3cd4f10" containerName="revoke-keys" Nov 03 15:25:26.445513 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:26.445489 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="382b7b36-beb8-4b8d-aa48-4071c3cd4f10" containerName="revoke-keys" Nov 03 15:25:26.448669 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:26.448654 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc" Nov 03 15:25:26.506788 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:26.506755 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc"] Nov 03 15:25:26.553183 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:26.553154 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:25:26.553329 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:26.553238 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:25:26.564383 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:26.564354 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fcf8a3-543c-4728-bc62-26a2385561a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eae949b-97zlc\" (UID: \"10fcf8a3-543c-4728-bc62-26a2385561a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc" Nov 03 15:25:26.564472 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:26.564404 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fcf8a3-543c-4728-bc62-26a2385561a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eae949b-97zlc\" (UID: \"10fcf8a3-543c-4728-bc62-26a2385561a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc" Nov 03 15:25:26.564548 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:26.564531 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rldfs\" (UniqueName: \"kubernetes.io/projected/10fcf8a3-543c-4728-bc62-26a2385561a4-kube-api-access-rldfs\") pod \"maas-api-revoke-sub-4eae949b-97zlc\" (UID: \"10fcf8a3-543c-4728-bc62-26a2385561a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc" Nov 03 15:25:26.665343 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:26.665310 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rldfs\" (UniqueName: \"kubernetes.io/projected/10fcf8a3-543c-4728-bc62-26a2385561a4-kube-api-access-rldfs\") pod \"maas-api-revoke-sub-4eae949b-97zlc\" (UID: \"10fcf8a3-543c-4728-bc62-26a2385561a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc" Nov 03 15:25:26.665495 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:26.665386 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fcf8a3-543c-4728-bc62-26a2385561a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eae949b-97zlc\" (UID: \"10fcf8a3-543c-4728-bc62-26a2385561a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc" Nov 03 15:25:26.665495 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:26.665422 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fcf8a3-543c-4728-bc62-26a2385561a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eae949b-97zlc\" (UID: \"10fcf8a3-543c-4728-bc62-26a2385561a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc" Nov 03 15:25:26.666149 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:26.666131 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fcf8a3-543c-4728-bc62-26a2385561a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eae949b-97zlc\" (UID: \"10fcf8a3-543c-4728-bc62-26a2385561a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc" Nov 03 15:25:26.667737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:26.667719 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fcf8a3-543c-4728-bc62-26a2385561a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eae949b-97zlc\" (UID: \"10fcf8a3-543c-4728-bc62-26a2385561a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc" Nov 03 15:25:26.727559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:26.727494 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rldfs\" (UniqueName: \"kubernetes.io/projected/10fcf8a3-543c-4728-bc62-26a2385561a4-kube-api-access-rldfs\") pod \"maas-api-revoke-sub-4eae949b-97zlc\" (UID: \"10fcf8a3-543c-4728-bc62-26a2385561a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc" Nov 03 15:25:26.761471 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:26.761444 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc" Nov 03 15:25:27.134608 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:27.134567 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc"] Nov 03 15:25:27.135301 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:25:27.135271 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10fcf8a3_543c_4728_bc62_26a2385561a4.slice/crio-4177582a0f911eeda2f1b24cd8b89235b824ff73e007442061bc0b8e508e8c83 WatchSource:0}: Error finding container 4177582a0f911eeda2f1b24cd8b89235b824ff73e007442061bc0b8e508e8c83: Status 404 returned error can't find the container with id 4177582a0f911eeda2f1b24cd8b89235b824ff73e007442061bc0b8e508e8c83 Nov 03 15:25:27.912091 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:27.912058 4557 generic.go:356] "Generic (PLEG): container finished" podID="10fcf8a3-543c-4728-bc62-26a2385561a4" containerID="4ecdb6d403f014ea9e8fec04883d5f50e7947643364bd54ded92c60567dc0788" exitCode=0 Nov 03 15:25:27.912507 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:27.912121 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc" event={"ID":"10fcf8a3-543c-4728-bc62-26a2385561a4","Type":"ContainerDied","Data":"4ecdb6d403f014ea9e8fec04883d5f50e7947643364bd54ded92c60567dc0788"} Nov 03 15:25:27.912507 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:27.912171 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc" event={"ID":"10fcf8a3-543c-4728-bc62-26a2385561a4","Type":"ContainerStarted","Data":"4177582a0f911eeda2f1b24cd8b89235b824ff73e007442061bc0b8e508e8c83"} Nov 03 15:25:28.784541 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.784509 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj"] Nov 03 15:25:28.788428 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.788413 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.841594 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.841566 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-34e8c5-openshift-default-dockercfg-f8ldx\"" Nov 03 15:25:28.889226 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.889198 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.889370 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.889230 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.889370 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.889256 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e9b467-83f6-4526-b8ae-2723410e32a3-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.889370 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.889343 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.889500 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.889443 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.889500 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.889479 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.889576 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.889528 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.889576 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.889547 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smw7x\" (UniqueName: \"kubernetes.io/projected/18e9b467-83f6-4526-b8ae-2723410e32a3-kube-api-access-smw7x\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.889689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.889621 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.894301 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.894280 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj"] Nov 03 15:25:28.990655 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.990614 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.991070 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.990681 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.991070 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.990717 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.991070 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.990747 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smw7x\" (UniqueName: \"kubernetes.io/projected/18e9b467-83f6-4526-b8ae-2723410e32a3-kube-api-access-smw7x\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.991070 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.990789 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.991070 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.990861 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.991070 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.990891 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.991070 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.990930 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e9b467-83f6-4526-b8ae-2723410e32a3-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.991070 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.990962 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.991510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.991279 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.991510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.991350 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.991510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.991414 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.991700 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.991547 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.992446 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.992419 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e9b467-83f6-4526-b8ae-2723410e32a3-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.993342 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.993316 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:28.993678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:28.993658 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:29.056879 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.056850 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:29.057002 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.056968 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smw7x\" (UniqueName: \"kubernetes.io/projected/18e9b467-83f6-4526-b8ae-2723410e32a3-kube-api-access-smw7x\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:29.060902 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.060885 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc" Nov 03 15:25:29.104032 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.104005 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:29.193402 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.192361 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10fcf8a3-543c-4728-bc62-26a2385561a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fcf8a3-543c-4728-bc62-26a2385561a4-maas-api-xks-ca\") pod \"10fcf8a3-543c-4728-bc62-26a2385561a4\" (UID: \"10fcf8a3-543c-4728-bc62-26a2385561a4\") " Nov 03 15:25:29.193402 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.192601 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10fcf8a3-543c-4728-bc62-26a2385561a4-kube-api-access-rldfs\" (UniqueName: \"kubernetes.io/projected/10fcf8a3-543c-4728-bc62-26a2385561a4-kube-api-access-rldfs\") pod \"10fcf8a3-543c-4728-bc62-26a2385561a4\" (UID: \"10fcf8a3-543c-4728-bc62-26a2385561a4\") " Nov 03 15:25:29.193402 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.192644 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10fcf8a3-543c-4728-bc62-26a2385561a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fcf8a3-543c-4728-bc62-26a2385561a4-maas-api-openshift-ca\") pod \"10fcf8a3-543c-4728-bc62-26a2385561a4\" (UID: \"10fcf8a3-543c-4728-bc62-26a2385561a4\") " Nov 03 15:25:29.193846 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.193816 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10fcf8a3-543c-4728-bc62-26a2385561a4-maas-api-openshift-ca" pod "10fcf8a3-543c-4728-bc62-26a2385561a4" (UID: "10fcf8a3-543c-4728-bc62-26a2385561a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:25:29.195916 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.195886 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10fcf8a3-543c-4728-bc62-26a2385561a4-kube-api-access-rldfs" pod "10fcf8a3-543c-4728-bc62-26a2385561a4" (UID: "10fcf8a3-543c-4728-bc62-26a2385561a4"). InnerVolumeSpecName "kube-api-access-rldfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:25:29.196006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.195982 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10fcf8a3-543c-4728-bc62-26a2385561a4-maas-api-xks-ca" pod "10fcf8a3-543c-4728-bc62-26a2385561a4" (UID: "10fcf8a3-543c-4728-bc62-26a2385561a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:25:29.293727 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.293656 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rldfs\" (UniqueName: \"kubernetes.io/projected/10fcf8a3-543c-4728-bc62-26a2385561a4-kube-api-access-rldfs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:29.293727 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.293679 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fcf8a3-543c-4728-bc62-26a2385561a4-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:29.293727 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.293691 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fcf8a3-543c-4728-bc62-26a2385561a4-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:29.297320 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.297291 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r"] Nov 03 15:25:29.297715 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.297700 4557 cpu_manager.go:414] "containerMap: removing container" podUID="10fcf8a3-543c-4728-bc62-26a2385561a4" containerName="revoke-keys" Nov 03 15:25:29.297715 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.297714 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10fcf8a3-543c-4728-bc62-26a2385561a4" containerName="revoke-keys" Nov 03 15:25:29.297867 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.297803 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="10fcf8a3-543c-4728-bc62-26a2385561a4" containerName="revoke-keys" Nov 03 15:25:29.302184 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.302162 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r" Nov 03 15:25:29.360114 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.360082 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r"] Nov 03 15:25:29.394424 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.394385 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5789edb-kpn7r\" (UID: \"d31fb2e6-c807-45ca-8800-9513f0a5b8e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r" Nov 03 15:25:29.394628 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.394433 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwwm9\" (UniqueName: \"kubernetes.io/projected/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-kube-api-access-hwwm9\") pod \"maas-api-revoke-sub-f5789edb-kpn7r\" (UID: \"d31fb2e6-c807-45ca-8800-9513f0a5b8e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r" Nov 03 15:25:29.394770 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.394745 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5789edb-kpn7r\" (UID: \"d31fb2e6-c807-45ca-8800-9513f0a5b8e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r" Nov 03 15:25:29.496352 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.496319 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5789edb-kpn7r\" (UID: \"d31fb2e6-c807-45ca-8800-9513f0a5b8e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r" Nov 03 15:25:29.496520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.496367 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwwm9\" (UniqueName: \"kubernetes.io/projected/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-kube-api-access-hwwm9\") pod \"maas-api-revoke-sub-f5789edb-kpn7r\" (UID: \"d31fb2e6-c807-45ca-8800-9513f0a5b8e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r" Nov 03 15:25:29.496520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.496418 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5789edb-kpn7r\" (UID: \"d31fb2e6-c807-45ca-8800-9513f0a5b8e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r" Nov 03 15:25:29.496971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.496948 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5789edb-kpn7r\" (UID: \"d31fb2e6-c807-45ca-8800-9513f0a5b8e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r" Nov 03 15:25:29.498650 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.498630 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5789edb-kpn7r\" (UID: \"d31fb2e6-c807-45ca-8800-9513f0a5b8e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r" Nov 03 15:25:29.532920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.532893 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj"] Nov 03 15:25:29.533534 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:25:29.533512 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e9b467_83f6_4526_b8ae_2723410e32a3.slice/crio-f498b6cdcf0474592e933bf4a885980ac09c51d6bcecd1534d1061461976d592 WatchSource:0}: Error finding container f498b6cdcf0474592e933bf4a885980ac09c51d6bcecd1534d1061461976d592: Status 404 returned error can't find the container with id f498b6cdcf0474592e933bf4a885980ac09c51d6bcecd1534d1061461976d592 Nov 03 15:25:29.535789 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.535764 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:25:29.535864 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.535819 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:25:29.535864 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.535857 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:25:29.557915 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.557893 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwwm9\" (UniqueName: \"kubernetes.io/projected/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-kube-api-access-hwwm9\") pod \"maas-api-revoke-sub-f5789edb-kpn7r\" (UID: \"d31fb2e6-c807-45ca-8800-9513f0a5b8e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r" Nov 03 15:25:29.618417 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.618388 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r" Nov 03 15:25:29.796134 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.796050 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:25:29.932266 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.932179 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" event={"ID":"18e9b467-83f6-4526-b8ae-2723410e32a3","Type":"ContainerStarted","Data":"5a22d594b4b76ecf3a0f3d919ca55a19b06a6f8fe7d762ed083faf9cac2f44af"} Nov 03 15:25:29.932266 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.932234 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" event={"ID":"18e9b467-83f6-4526-b8ae-2723410e32a3","Type":"ContainerStarted","Data":"f498b6cdcf0474592e933bf4a885980ac09c51d6bcecd1534d1061461976d592"} Nov 03 15:25:29.933729 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.933704 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc" event={"ID":"10fcf8a3-543c-4728-bc62-26a2385561a4","Type":"ContainerDied","Data":"4177582a0f911eeda2f1b24cd8b89235b824ff73e007442061bc0b8e508e8c83"} Nov 03 15:25:29.933868 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.933740 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc" Nov 03 15:25:29.933868 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:29.933742 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4177582a0f911eeda2f1b24cd8b89235b824ff73e007442061bc0b8e508e8c83" Nov 03 15:25:30.000212 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:25:30.000184 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31fb2e6_c807_45ca_8800_9513f0a5b8e1.slice/crio-d38fc55165676c113bce765b6ca787d920f24e01de151665d9954f00acf2ad36 WatchSource:0}: Error finding container d38fc55165676c113bce765b6ca787d920f24e01de151665d9954f00acf2ad36: Status 404 returned error can't find the container with id d38fc55165676c113bce765b6ca787d920f24e01de151665d9954f00acf2ad36 Nov 03 15:25:30.001506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:30.001486 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r"] Nov 03 15:25:30.054385 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:30.054340 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" podStartSLOduration=2.054327449 podStartE2EDuration="2.054327449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:30.053175222 +0000 UTC m=+1039.896296763" watchObservedRunningTime="2026-11-03 15:25:30.054327449 +0000 UTC m=+1039.897448991" Nov 03 15:25:30.104393 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:30.104369 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:30.105633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:30.105611 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 15:25:30.105729 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:30.105657 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" podUID="18e9b467-83f6-4526-b8ae-2723410e32a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 15:25:30.942102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:30.942071 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r" event={"ID":"d31fb2e6-c807-45ca-8800-9513f0a5b8e1","Type":"ContainerStarted","Data":"eb5545a6cedb5da19825d6ef1eb93f6bddec50cc13b13d6c4370dd302b867fe8"} Nov 03 15:25:30.942297 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:30.942125 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r" event={"ID":"d31fb2e6-c807-45ca-8800-9513f0a5b8e1","Type":"ContainerStarted","Data":"d38fc55165676c113bce765b6ca787d920f24e01de151665d9954f00acf2ad36"} Nov 03 15:25:31.105276 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:31.105241 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 15:25:31.105746 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:31.105305 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" podUID="18e9b467-83f6-4526-b8ae-2723410e32a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 15:25:31.128065 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:31.128013 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r" podStartSLOduration=2.128000412 podStartE2EDuration="2.128000412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:31.127540537 +0000 UTC m=+1040.970662081" watchObservedRunningTime="2026-11-03 15:25:31.128000412 +0000 UTC m=+1040.971122008" Nov 03 15:25:31.949452 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:31.949413 4557 generic.go:356] "Generic (PLEG): container finished" podID="d31fb2e6-c807-45ca-8800-9513f0a5b8e1" containerID="eb5545a6cedb5da19825d6ef1eb93f6bddec50cc13b13d6c4370dd302b867fe8" exitCode=0 Nov 03 15:25:31.949649 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:31.949487 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r" event={"ID":"d31fb2e6-c807-45ca-8800-9513f0a5b8e1","Type":"ContainerDied","Data":"eb5545a6cedb5da19825d6ef1eb93f6bddec50cc13b13d6c4370dd302b867fe8"} Nov 03 15:25:32.105381 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:32.105346 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 15:25:32.105886 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:32.105666 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" podUID="18e9b467-83f6-4526-b8ae-2723410e32a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 15:25:33.113607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:33.105320 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 15:25:33.113607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:33.105407 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" podUID="18e9b467-83f6-4526-b8ae-2723410e32a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 15:25:33.207901 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:33.207825 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r" Nov 03 15:25:33.347519 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:33.347488 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-maas-api-xks-ca\") pod \"d31fb2e6-c807-45ca-8800-9513f0a5b8e1\" (UID: \"d31fb2e6-c807-45ca-8800-9513f0a5b8e1\") " Nov 03 15:25:33.347733 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:33.347558 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-maas-api-openshift-ca\") pod \"d31fb2e6-c807-45ca-8800-9513f0a5b8e1\" (UID: \"d31fb2e6-c807-45ca-8800-9513f0a5b8e1\") " Nov 03 15:25:33.347882 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:33.347859 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-kube-api-access-hwwm9\" (UniqueName: \"kubernetes.io/projected/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-kube-api-access-hwwm9\") pod \"d31fb2e6-c807-45ca-8800-9513f0a5b8e1\" (UID: \"d31fb2e6-c807-45ca-8800-9513f0a5b8e1\") " Nov 03 15:25:33.348107 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:33.347925 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-maas-api-openshift-ca" pod "d31fb2e6-c807-45ca-8800-9513f0a5b8e1" (UID: "d31fb2e6-c807-45ca-8800-9513f0a5b8e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:25:33.348413 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:33.348393 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:33.350104 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:33.349987 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-maas-api-xks-ca" pod "d31fb2e6-c807-45ca-8800-9513f0a5b8e1" (UID: "d31fb2e6-c807-45ca-8800-9513f0a5b8e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:25:33.350104 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:33.350073 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-kube-api-access-hwwm9" pod "d31fb2e6-c807-45ca-8800-9513f0a5b8e1" (UID: "d31fb2e6-c807-45ca-8800-9513f0a5b8e1"). InnerVolumeSpecName "kube-api-access-hwwm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:25:33.448989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:33.448953 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:33.448989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:33.448987 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwwm9\" (UniqueName: \"kubernetes.io/projected/d31fb2e6-c807-45ca-8800-9513f0a5b8e1-kube-api-access-hwwm9\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:25:33.964997 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:33.964969 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r" Nov 03 15:25:33.965162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:33.965050 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r" event={"ID":"d31fb2e6-c807-45ca-8800-9513f0a5b8e1","Type":"ContainerDied","Data":"d38fc55165676c113bce765b6ca787d920f24e01de151665d9954f00acf2ad36"} Nov 03 15:25:33.965162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:33.965100 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d38fc55165676c113bce765b6ca787d920f24e01de151665d9954f00acf2ad36" Nov 03 15:25:34.105220 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:34.105186 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 15:25:34.105386 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:34.105240 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" podUID="18e9b467-83f6-4526-b8ae-2723410e32a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 15:25:34.857471 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:34.857437 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:25:35.105065 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:35.105027 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 15:25:35.105375 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:35.105340 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" podUID="18e9b467-83f6-4526-b8ae-2723410e32a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 15:25:36.105447 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:36.105410 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 15:25:36.105966 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:36.105474 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" podUID="18e9b467-83f6-4526-b8ae-2723410e32a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 15:25:36.464769 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:36.464686 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:25:37.243143 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:37.243111 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:37.243812 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:37.243790 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:37.244505 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:37.244485 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:25:39.911131 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:39.911086 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:25:41.256546 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:41.256483 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:25:44.949045 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:44.949002 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:25:46.530141 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:25:46.530099 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:25:50.062927 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:50.062841 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:25:55.125973 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:55.125935 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:25:56.772046 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.772016 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45"] Nov 03 15:25:56.772455 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.772442 4557 cpu_manager.go:414] "containerMap: removing container" podUID="d31fb2e6-c807-45ca-8800-9513f0a5b8e1" containerName="revoke-keys" Nov 03 15:25:56.772455 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.772455 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d31fb2e6-c807-45ca-8800-9513f0a5b8e1" containerName="revoke-keys" Nov 03 15:25:56.772563 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.772534 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="d31fb2e6-c807-45ca-8800-9513f0a5b8e1" containerName="revoke-keys" Nov 03 15:25:56.776945 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.776928 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:56.831114 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.831081 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f59d26\"/\"distinct-w6-f59d26-kserve-self-signed-certs\"" Nov 03 15:25:56.834463 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.834431 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45"] Nov 03 15:25:56.885204 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.885125 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:25:56.897531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.897491 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-model-cache\") pod \"distinct-w6-f59d26-kserve-6c6cd598d7-6kb45\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:56.897531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.897529 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-dshm\") pod \"distinct-w6-f59d26-kserve-6c6cd598d7-6kb45\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:56.897746 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.897548 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8499b913-2f91-43f1-bede-5a7462494ef8-tls-certs\") pod \"distinct-w6-f59d26-kserve-6c6cd598d7-6kb45\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:56.897806 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.897786 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mp4t\" (UniqueName: \"kubernetes.io/projected/8499b913-2f91-43f1-bede-5a7462494ef8-kube-api-access-9mp4t\") pod \"distinct-w6-f59d26-kserve-6c6cd598d7-6kb45\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:56.897855 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.897838 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-tmp-dir\") pod \"distinct-w6-f59d26-kserve-6c6cd598d7-6kb45\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:56.897958 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.897936 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-home\") pod \"distinct-w6-f59d26-kserve-6c6cd598d7-6kb45\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:56.999490 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.999442 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-tmp-dir\") pod \"distinct-w6-f59d26-kserve-6c6cd598d7-6kb45\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:56.999490 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.999494 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-home\") pod \"distinct-w6-f59d26-kserve-6c6cd598d7-6kb45\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:56.999764 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.999532 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-model-cache\") pod \"distinct-w6-f59d26-kserve-6c6cd598d7-6kb45\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:56.999764 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.999549 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-dshm\") pod \"distinct-w6-f59d26-kserve-6c6cd598d7-6kb45\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:56.999764 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.999567 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8499b913-2f91-43f1-bede-5a7462494ef8-tls-certs\") pod \"distinct-w6-f59d26-kserve-6c6cd598d7-6kb45\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:56.999764 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.999715 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mp4t\" (UniqueName: \"kubernetes.io/projected/8499b913-2f91-43f1-bede-5a7462494ef8-kube-api-access-9mp4t\") pod \"distinct-w6-f59d26-kserve-6c6cd598d7-6kb45\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:56.999985 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.999938 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-tmp-dir\") pod \"distinct-w6-f59d26-kserve-6c6cd598d7-6kb45\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:57.000042 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:56.999987 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-home\") pod \"distinct-w6-f59d26-kserve-6c6cd598d7-6kb45\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:57.000098 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:57.000038 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-model-cache\") pod \"distinct-w6-f59d26-kserve-6c6cd598d7-6kb45\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:57.001911 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:57.001877 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-dshm\") pod \"distinct-w6-f59d26-kserve-6c6cd598d7-6kb45\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:57.002158 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:57.002138 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8499b913-2f91-43f1-bede-5a7462494ef8-tls-certs\") pod \"distinct-w6-f59d26-kserve-6c6cd598d7-6kb45\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:57.061870 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:57.061837 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mp4t\" (UniqueName: \"kubernetes.io/projected/8499b913-2f91-43f1-bede-5a7462494ef8-kube-api-access-9mp4t\") pod \"distinct-w6-f59d26-kserve-6c6cd598d7-6kb45\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:57.090956 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:57.090918 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:57.511355 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:57.511321 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45"] Nov 03 15:25:57.511706 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:25:57.511684 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8499b913_2f91_43f1_bede_5a7462494ef8.slice/crio-4645221e2cd614642daf8438fefdc2381f9295671866205d67e8fbde3e46fbc8 WatchSource:0}: Error finding container 4645221e2cd614642daf8438fefdc2381f9295671866205d67e8fbde3e46fbc8: Status 404 returned error can't find the container with id 4645221e2cd614642daf8438fefdc2381f9295671866205d67e8fbde3e46fbc8 Nov 03 15:25:58.106688 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:58.106648 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" event={"ID":"8499b913-2f91-43f1-bede-5a7462494ef8","Type":"ContainerStarted","Data":"9641131ce77b6edaabf2ee148f3d4857eaf0a2e170668e9ab221df8baf2e7ebf"} Nov 03 15:25:58.106688 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:58.106689 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" event={"ID":"8499b913-2f91-43f1-bede-5a7462494ef8","Type":"ContainerStarted","Data":"4645221e2cd614642daf8438fefdc2381f9295671866205d67e8fbde3e46fbc8"} Nov 03 15:25:58.224745 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:58.224690 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podStartSLOduration=2.224676123 podStartE2EDuration="2.224676123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:58.224137404 +0000 UTC m=+1068.067258947" watchObservedRunningTime="2026-11-03 15:25:58.224676123 +0000 UTC m=+1068.067797665" Nov 03 15:25:59.091555 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:59.091521 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:25:59.936230 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:59.936193 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb"] Nov 03 15:25:59.940439 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:25:59.940421 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.035225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.031286 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.035225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.031339 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.035225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.031384 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.035225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.031409 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.035225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.031441 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.035225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.031472 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.035225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.031512 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w4fk\" (UniqueName: \"kubernetes.io/projected/5dc6676c-58c3-4ad9-9701-e76f70622bb8-kube-api-access-4w4fk\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.035225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.031594 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.035225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.031638 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.047625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.046335 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:26:00.132822 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.132795 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.133008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.132833 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.133008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.132888 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.133008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.132908 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.133008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.132940 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.133008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.132963 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.133008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.132992 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.133276 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.133035 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.133276 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.133076 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w4fk\" (UniqueName: \"kubernetes.io/projected/5dc6676c-58c3-4ad9-9701-e76f70622bb8-kube-api-access-4w4fk\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.133476 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.133449 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.133687 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.133628 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.133687 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.133512 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.133876 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.133757 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.134010 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.133983 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.135232 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.135211 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.135369 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.135350 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.155452 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.155430 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb"] Nov 03 15:26:00.197269 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.197214 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w4fk\" (UniqueName: \"kubernetes.io/projected/5dc6676c-58c3-4ad9-9701-e76f70622bb8-kube-api-access-4w4fk\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.197269 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.197258 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.254399 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.254372 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:00.693394 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.693367 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb"] Nov 03 15:26:00.693800 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:26:00.693776 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc6676c_58c3_4ad9_9701_e76f70622bb8.slice/crio-c5ad62ee8ad2b9eef752fa215d7a4f09a6e4661ae98d91c6617630854a967c29 WatchSource:0}: Error finding container c5ad62ee8ad2b9eef752fa215d7a4f09a6e4661ae98d91c6617630854a967c29: Status 404 returned error can't find the container with id c5ad62ee8ad2b9eef752fa215d7a4f09a6e4661ae98d91c6617630854a967c29 Nov 03 15:26:00.695990 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.695961 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:26:00.696059 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.696021 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:26:00.696059 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:00.696048 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:26:01.055683 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:01.055649 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:26:01.132205 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:01.131983 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" event={"ID":"5dc6676c-58c3-4ad9-9701-e76f70622bb8","Type":"ContainerStarted","Data":"25465e3e1fa3eb61806c2665563ef485207842578c7d0f744517f19ae7f7d875"} Nov 03 15:26:01.132205 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:01.132055 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" event={"ID":"5dc6676c-58c3-4ad9-9701-e76f70622bb8","Type":"ContainerStarted","Data":"c5ad62ee8ad2b9eef752fa215d7a4f09a6e4661ae98d91c6617630854a967c29"} Nov 03 15:26:01.248130 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:01.248079 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:26:01.254973 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:01.254946 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:01.256677 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:01.256652 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 15:26:01.256795 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:01.256708 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" podUID="5dc6676c-58c3-4ad9-9701-e76f70622bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 15:26:01.261153 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:01.261100 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" podStartSLOduration=2.261084261 podStartE2EDuration="2.261084261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:26:01.257711604 +0000 UTC m=+1071.100833165" watchObservedRunningTime="2026-11-03 15:26:01.261084261 +0000 UTC m=+1071.104205806" Nov 03 15:26:02.255195 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:02.255162 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 15:26:02.255630 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:02.255214 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" podUID="5dc6676c-58c3-4ad9-9701-e76f70622bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 15:26:03.255752 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:03.255665 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 15:26:03.255752 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:03.255727 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" podUID="5dc6676c-58c3-4ad9-9701-e76f70622bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 15:26:04.255711 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:04.255673 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 15:26:04.255898 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:04.255731 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" podUID="5dc6676c-58c3-4ad9-9701-e76f70622bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 15:26:04.893612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:04.893487 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:26:05.255194 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:05.255113 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 15:26:05.255194 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:05.255176 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" podUID="5dc6676c-58c3-4ad9-9701-e76f70622bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 15:26:06.255443 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:06.255407 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 15:26:06.255877 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:06.255469 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" podUID="5dc6676c-58c3-4ad9-9701-e76f70622bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 15:26:07.091090 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:07.091054 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:26:07.103664 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:07.103639 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:26:07.190604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:07.189832 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:26:07.255151 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:07.255117 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 15:26:07.255325 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:07.255182 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" podUID="5dc6676c-58c3-4ad9-9701-e76f70622bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 15:26:08.259786 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:08.259754 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:08.260279 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:08.260137 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:08.260703 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:08.260685 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:26:09.817153 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:09.817117 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:26:14.715778 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:14.715741 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:26:16.529245 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:26:16.529213 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:26:19.613527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:19.613492 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:26:24.543838 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:24.543798 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:26:26.172099 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:26.172052 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:26:29.520937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:29.520903 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:26:34.520758 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:34.520711 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:26:39.402189 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:39.402153 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:26:44.298550 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:44.298509 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:26:44.942759 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:44.942725 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg"] Nov 03 15:26:44.947036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:44.947020 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.001814 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.001790 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-9855ff-openshift-default-dockercfg-5hjbk\"" Nov 03 15:26:45.057021 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.056994 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.057174 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.057047 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.057174 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.057108 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg"] Nov 03 15:26:45.057316 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.057176 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.057316 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.057238 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnmdk\" (UniqueName: \"kubernetes.io/projected/b37f9ec1-b784-41fd-9a79-8df424aa77e5-kube-api-access-dnmdk\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.057316 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.057309 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.057469 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.057342 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.057469 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.057378 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.057469 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.057402 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.057597 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.057467 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.158351 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.158317 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.158520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.158356 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.158520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.158479 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.158520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.158511 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.158716 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.158539 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.158716 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.158653 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.158832 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.158741 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.158832 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.158747 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.158832 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.158824 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.158978 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.158870 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.158978 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.158891 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.158978 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.158949 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnmdk\" (UniqueName: \"kubernetes.io/projected/b37f9ec1-b784-41fd-9a79-8df424aa77e5-kube-api-access-dnmdk\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.159135 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.159100 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.159373 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.159352 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.160685 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.160661 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.160988 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.160970 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.222645 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.222556 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnmdk\" (UniqueName: \"kubernetes.io/projected/b37f9ec1-b784-41fd-9a79-8df424aa77e5-kube-api-access-dnmdk\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.225678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.225656 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.259873 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.259853 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:45.693857 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:26:45.693810 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37f9ec1_b784_41fd_9a79_8df424aa77e5.slice/crio-d8330a1446404bd80493b50dc016dba969f9513ba615f0460eea934021882fae WatchSource:0}: Error finding container d8330a1446404bd80493b50dc016dba969f9513ba615f0460eea934021882fae: Status 404 returned error can't find the container with id d8330a1446404bd80493b50dc016dba969f9513ba615f0460eea934021882fae Nov 03 15:26:45.695695 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.695670 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg"] Nov 03 15:26:45.696181 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.696154 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:26:45.696226 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.696213 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:26:45.696263 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:45.696240 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:26:46.412141 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:46.412109 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" event={"ID":"b37f9ec1-b784-41fd-9a79-8df424aa77e5","Type":"ContainerStarted","Data":"b7c46f81e707c8a068eeac7090cbca8557434d8e002d0dab988d451bd5ef8c7d"} Nov 03 15:26:46.412325 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:46.412168 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" event={"ID":"b37f9ec1-b784-41fd-9a79-8df424aa77e5","Type":"ContainerStarted","Data":"d8330a1446404bd80493b50dc016dba969f9513ba615f0460eea934021882fae"} Nov 03 15:26:46.530622 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:26:46.530560 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:26:47.260725 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.260693 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:47.262228 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.262200 4557 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 15:26:47.262371 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.262255 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" podUID="b37f9ec1-b784-41fd-9a79-8df424aa77e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 15:26:47.357026 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.356969 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" podStartSLOduration=3.356955026 podStartE2EDuration="3.356955026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:26:46.618752016 +0000 UTC m=+1116.461873557" watchObservedRunningTime="2026-11-03 15:26:47.356955026 +0000 UTC m=+1117.200076568" Nov 03 15:26:47.359920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.359899 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx"] Nov 03 15:26:47.363813 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.363792 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.397242 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.397205 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-dshm\") pod \"distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.397448 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.397426 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-model-cache\") pod \"distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.397606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.397569 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-home\") pod \"distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.397762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.397741 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-tmp-dir\") pod \"distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.405607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.397951 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4mwm\" (UniqueName: \"kubernetes.io/projected/62d54cd1-cc1f-488b-9702-739043f4753c-kube-api-access-r4mwm\") pod \"distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.405847 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.405824 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d54cd1-cc1f-488b-9702-739043f4753c-tls-certs\") pod \"distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.427574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.427550 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx"] Nov 03 15:26:47.430995 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.430965 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f78fd6\"/\"distinct-2-w2-f78fd6-kserve-self-signed-certs\"" Nov 03 15:26:47.511515 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.511421 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4mwm\" (UniqueName: \"kubernetes.io/projected/62d54cd1-cc1f-488b-9702-739043f4753c-kube-api-access-r4mwm\") pod \"distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.512171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.512138 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d54cd1-cc1f-488b-9702-739043f4753c-tls-certs\") pod \"distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.512406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.512384 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-dshm\") pod \"distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.512495 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.512422 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-model-cache\") pod \"distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.512495 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.512474 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-home\") pod \"distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.512617 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.512530 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-tmp-dir\") pod \"distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.513100 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.513076 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:26:47.513260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.513112 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-tmp-dir\") pod \"distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.513467 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.513446 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-model-cache\") pod \"distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.513735 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.513714 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-home\") pod \"distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.529333 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.529304 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d54cd1-cc1f-488b-9702-739043f4753c-tls-certs\") pod \"distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.531520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.531278 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-dshm\") pod \"distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.573425 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.573393 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4mwm\" (UniqueName: \"kubernetes.io/projected/62d54cd1-cc1f-488b-9702-739043f4753c-kube-api-access-r4mwm\") pod \"distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:47.676785 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:47.676748 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:48.165195 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:48.165168 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx"] Nov 03 15:26:48.165508 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:26:48.165483 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62d54cd1_cc1f_488b_9702_739043f4753c.slice/crio-8b7ac791d73a3ebc829be37934a2fe9504a675d26b52e901634765ecb66b7004 WatchSource:0}: Error finding container 8b7ac791d73a3ebc829be37934a2fe9504a675d26b52e901634765ecb66b7004: Status 404 returned error can't find the container with id 8b7ac791d73a3ebc829be37934a2fe9504a675d26b52e901634765ecb66b7004 Nov 03 15:26:48.260310 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:48.260277 4557 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 15:26:48.260407 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:48.260319 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" podUID="b37f9ec1-b784-41fd-9a79-8df424aa77e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 15:26:48.427369 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:48.427274 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" event={"ID":"62d54cd1-cc1f-488b-9702-739043f4753c","Type":"ContainerStarted","Data":"53792f4acb891e1b7664d7c520992cb12849d6edd7ead71cb8362e3bd3bc7485"} Nov 03 15:26:48.427369 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:48.427320 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" event={"ID":"62d54cd1-cc1f-488b-9702-739043f4753c","Type":"ContainerStarted","Data":"8b7ac791d73a3ebc829be37934a2fe9504a675d26b52e901634765ecb66b7004"} Nov 03 15:26:48.692828 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:48.692743 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:49.214790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:49.214745 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:26:49.260318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:49.260287 4557 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 15:26:49.260511 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:49.260344 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" podUID="b37f9ec1-b784-41fd-9a79-8df424aa77e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 15:26:50.260932 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:50.260898 4557 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 15:26:50.261504 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:50.260946 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" podUID="b37f9ec1-b784-41fd-9a79-8df424aa77e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 15:26:51.260559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:51.260520 4557 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 15:26:51.260773 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:51.260598 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" podUID="b37f9ec1-b784-41fd-9a79-8df424aa77e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 15:26:51.547015 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:51.546981 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:26:52.260884 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:52.260847 4557 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 15:26:52.261062 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:52.260912 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" podUID="b37f9ec1-b784-41fd-9a79-8df424aa77e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 15:26:53.264780 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:53.264700 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:53.265177 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:53.265034 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:53.265809 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:53.265785 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:26:53.437937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:53.437867 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podStartSLOduration=6.437852839 podStartE2EDuration="6.437852839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:26:48.564084917 +0000 UTC m=+1118.407206456" watchObservedRunningTime="2026-11-03 15:26:53.437852839 +0000 UTC m=+1123.280974380" Nov 03 15:26:54.148638 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:54.148573 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:26:57.678219 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:57.678180 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:57.690591 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:57.690553 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:58.513721 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:58.513694 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:26:59.017946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:26:59.017914 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:27:01.574300 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:01.574264 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:27:03.920753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:03.920720 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:27:04.779361 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.774844 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg"] Nov 03 15:27:04.783277 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.783243 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:04.837747 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.837714 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg"] Nov 03 15:27:04.844804 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.844780 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-dshm\") pod \"distinct-2-w3-8b841a-kserve-76685757b9-fzrlg\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:04.844947 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.844909 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5256df6b-b047-4aae-82da-fba43402fbe0-tls-certs\") pod \"distinct-2-w3-8b841a-kserve-76685757b9-fzrlg\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:04.845018 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.844957 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-model-cache\") pod \"distinct-2-w3-8b841a-kserve-76685757b9-fzrlg\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:04.845018 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.845005 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-tmp-dir\") pod \"distinct-2-w3-8b841a-kserve-76685757b9-fzrlg\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:04.845142 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.845076 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-home\") pod \"distinct-2-w3-8b841a-kserve-76685757b9-fzrlg\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:04.845211 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.845190 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d7dg\" (UniqueName: \"kubernetes.io/projected/5256df6b-b047-4aae-82da-fba43402fbe0-kube-api-access-7d7dg\") pod \"distinct-2-w3-8b841a-kserve-76685757b9-fzrlg\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:04.883741 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.883712 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8b841a\"/\"default-dockercfg-sz8vc\"" Nov 03 15:27:04.883871 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.883752 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8b841a\"/\"distinct-2-w3-8b841a-kserve-self-signed-certs\"" Nov 03 15:27:04.886023 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.886008 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8b841a\"/\"kube-root-ca.crt\"" Nov 03 15:27:04.886173 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.886155 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8b841a\"/\"openshift-service-ca.crt\"" Nov 03 15:27:04.925852 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.925816 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:27:04.946061 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.946028 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5256df6b-b047-4aae-82da-fba43402fbe0-tls-certs\") pod \"distinct-2-w3-8b841a-kserve-76685757b9-fzrlg\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:04.946267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.946076 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-model-cache\") pod \"distinct-2-w3-8b841a-kserve-76685757b9-fzrlg\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:04.946267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.946110 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-tmp-dir\") pod \"distinct-2-w3-8b841a-kserve-76685757b9-fzrlg\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:04.946267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.946135 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-home\") pod \"distinct-2-w3-8b841a-kserve-76685757b9-fzrlg\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:04.946267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.946195 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d7dg\" (UniqueName: \"kubernetes.io/projected/5256df6b-b047-4aae-82da-fba43402fbe0-kube-api-access-7d7dg\") pod \"distinct-2-w3-8b841a-kserve-76685757b9-fzrlg\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:04.946267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.946261 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-dshm\") pod \"distinct-2-w3-8b841a-kserve-76685757b9-fzrlg\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:04.946641 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.946553 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-tmp-dir\") pod \"distinct-2-w3-8b841a-kserve-76685757b9-fzrlg\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:04.946641 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.946628 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-model-cache\") pod \"distinct-2-w3-8b841a-kserve-76685757b9-fzrlg\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:04.946741 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.946662 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-home\") pod \"distinct-2-w3-8b841a-kserve-76685757b9-fzrlg\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:04.948403 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.948382 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-dshm\") pod \"distinct-2-w3-8b841a-kserve-76685757b9-fzrlg\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:04.948702 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:04.948686 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5256df6b-b047-4aae-82da-fba43402fbe0-tls-certs\") pod \"distinct-2-w3-8b841a-kserve-76685757b9-fzrlg\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:05.008332 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:05.008299 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d7dg\" (UniqueName: \"kubernetes.io/projected/5256df6b-b047-4aae-82da-fba43402fbe0-kube-api-access-7d7dg\") pod \"distinct-2-w3-8b841a-kserve-76685757b9-fzrlg\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:05.096733 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:05.096700 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:05.490093 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:05.490067 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg"] Nov 03 15:27:05.490734 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:27:05.490710 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5256df6b_b047_4aae_82da_fba43402fbe0.slice/crio-9f298760d5491b2614bfcc84d3acb8a215f1f00527e8d091fc06c2ca134ae0d1 WatchSource:0}: Error finding container 9f298760d5491b2614bfcc84d3acb8a215f1f00527e8d091fc06c2ca134ae0d1: Status 404 returned error can't find the container with id 9f298760d5491b2614bfcc84d3acb8a215f1f00527e8d091fc06c2ca134ae0d1 Nov 03 15:27:05.542898 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:05.542870 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" event={"ID":"5256df6b-b047-4aae-82da-fba43402fbe0","Type":"ContainerStarted","Data":"9f298760d5491b2614bfcc84d3acb8a215f1f00527e8d091fc06c2ca134ae0d1"} Nov 03 15:27:06.550075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:06.550035 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" event={"ID":"5256df6b-b047-4aae-82da-fba43402fbe0","Type":"ContainerStarted","Data":"9dea0f430fee7361fdf2ad4457d4f1670dd7ffa674fa0c6f81229e423fb0038a"} Nov 03 15:27:06.671351 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:06.671295 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podStartSLOduration=2.671281023 podStartE2EDuration="2.671281023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:27:06.668928958 +0000 UTC m=+1136.512050502" watchObservedRunningTime="2026-11-03 15:27:06.671281023 +0000 UTC m=+1136.514402564" Nov 03 15:27:07.097769 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:07.097737 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:08.820915 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:08.820863 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:27:13.712570 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:13.712519 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:27:13.900319 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:13.900285 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:27:15.096988 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:15.096952 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:15.109799 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:15.109775 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:15.622600 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:15.622559 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:27:16.529463 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:27:16.529431 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:27:18.760852 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:18.760817 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:27:23.783452 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:23.783395 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:27:28.773676 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:28.773638 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:27:31.255199 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.255165 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm"] Nov 03 15:27:31.258884 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.258867 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.285659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.285631 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.285768 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.285668 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.285768 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.285689 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.285853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.285834 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7dqx\" (UniqueName: \"kubernetes.io/projected/72be94c1-dd49-4f71-8a37-6056e0939ee9-kube-api-access-w7dqx\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.285889 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.285874 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.285941 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.285925 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.285979 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.285963 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72be94c1-dd49-4f71-8a37-6056e0939ee9-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.286013 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.285986 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.286013 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.286002 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.311681 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.311657 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8b841a-openshift-default-dockercfg-bszcv\"" Nov 03 15:27:31.366678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.366647 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm"] Nov 03 15:27:31.386706 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.386673 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.386861 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.386718 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.386861 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.386752 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.386982 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.386865 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7dqx\" (UniqueName: \"kubernetes.io/projected/72be94c1-dd49-4f71-8a37-6056e0939ee9-kube-api-access-w7dqx\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.386982 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.386973 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.387090 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.387029 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.387147 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.387136 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72be94c1-dd49-4f71-8a37-6056e0939ee9-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.387225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.387206 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.387283 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.387238 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.387283 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.387261 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.387873 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.387850 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.388758 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.388738 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72be94c1-dd49-4f71-8a37-6056e0939ee9-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.388869 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.388849 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.389442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.389419 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.389674 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.389654 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.391051 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.391030 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.450290 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.450260 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7dqx\" (UniqueName: \"kubernetes.io/projected/72be94c1-dd49-4f71-8a37-6056e0939ee9-kube-api-access-w7dqx\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.471329 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.471294 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:31.571792 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:31.571763 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:32.015604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:32.015563 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm"] Nov 03 15:27:32.016752 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:27:32.016708 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72be94c1_dd49_4f71_8a37_6056e0939ee9.slice/crio-36ba4cc55b2e60e219e24d7ecda9032e875ec5fbb86f8e13c785a108e7eb014c WatchSource:0}: Error finding container 36ba4cc55b2e60e219e24d7ecda9032e875ec5fbb86f8e13c785a108e7eb014c: Status 404 returned error can't find the container with id 36ba4cc55b2e60e219e24d7ecda9032e875ec5fbb86f8e13c785a108e7eb014c Nov 03 15:27:32.019145 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:32.019114 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:27:32.019232 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:32.019174 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:27:32.019232 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:32.019204 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:27:32.737800 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:32.737765 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" event={"ID":"72be94c1-dd49-4f71-8a37-6056e0939ee9","Type":"ContainerStarted","Data":"8badc86a51cc816f5ac6db0185a2bb8f37c11cbe934663cfc33513b64d01fe91"} Nov 03 15:27:32.738183 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:32.737824 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" event={"ID":"72be94c1-dd49-4f71-8a37-6056e0939ee9","Type":"ContainerStarted","Data":"36ba4cc55b2e60e219e24d7ecda9032e875ec5fbb86f8e13c785a108e7eb014c"} Nov 03 15:27:32.970115 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:32.970059 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" podStartSLOduration=1.970043762 podStartE2EDuration="1.970043762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:27:32.91920576 +0000 UTC m=+1162.762327303" watchObservedRunningTime="2026-11-03 15:27:32.970043762 +0000 UTC m=+1162.813165303" Nov 03 15:27:33.572779 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:33.572742 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:33.574301 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:33.574279 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 15:27:33.574395 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:33.574338 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" podUID="72be94c1-dd49-4f71-8a37-6056e0939ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 15:27:33.605537 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:33.605501 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:27:34.572450 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:34.572414 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 15:27:34.572940 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:34.572477 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" podUID="72be94c1-dd49-4f71-8a37-6056e0939ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 15:27:35.572910 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:35.572873 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 15:27:35.573288 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:35.572947 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" podUID="72be94c1-dd49-4f71-8a37-6056e0939ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 15:27:36.572163 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:36.572128 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 15:27:36.572342 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:36.572189 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" podUID="72be94c1-dd49-4f71-8a37-6056e0939ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 15:27:37.572376 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:37.572340 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 15:27:37.572797 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:37.572405 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" podUID="72be94c1-dd49-4f71-8a37-6056e0939ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 15:27:38.493232 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:38.493202 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:27:38.572612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:38.572567 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 15:27:38.573089 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:38.572635 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" podUID="72be94c1-dd49-4f71-8a37-6056e0939ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 15:27:39.577385 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:39.577353 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:39.577874 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:39.577617 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:39.578301 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:39.578284 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:27:39.791640 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:39.791593 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:27:43.406690 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:43.406606 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:27:46.532652 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:27:46.532604 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:27:48.406898 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:48.406865 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:27:53.315739 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:53.315704 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc"] Nov 03 15:27:53.319734 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:53.319705 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc" Nov 03 15:27:53.347913 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:53.347879 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:27:53.379224 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:53.379190 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc"] Nov 03 15:27:53.423277 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:53.423238 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:27:53.423447 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:53.423287 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:27:53.428058 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:53.428029 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb55f\" (UniqueName: \"kubernetes.io/projected/09ef56ec-1f91-410b-8fda-53c02e439893-kube-api-access-rb55f\") pod \"maas-api-revoke-sub-4f550b8c-85mjc\" (UID: \"09ef56ec-1f91-410b-8fda-53c02e439893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc" Nov 03 15:27:53.428252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:53.428230 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ef56ec-1f91-410b-8fda-53c02e439893-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f550b8c-85mjc\" (UID: \"09ef56ec-1f91-410b-8fda-53c02e439893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc" Nov 03 15:27:53.428335 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:53.428318 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ef56ec-1f91-410b-8fda-53c02e439893-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f550b8c-85mjc\" (UID: \"09ef56ec-1f91-410b-8fda-53c02e439893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc" Nov 03 15:27:53.530202 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:53.530152 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ef56ec-1f91-410b-8fda-53c02e439893-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f550b8c-85mjc\" (UID: \"09ef56ec-1f91-410b-8fda-53c02e439893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc" Nov 03 15:27:53.530407 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:53.530274 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb55f\" (UniqueName: \"kubernetes.io/projected/09ef56ec-1f91-410b-8fda-53c02e439893-kube-api-access-rb55f\") pod \"maas-api-revoke-sub-4f550b8c-85mjc\" (UID: \"09ef56ec-1f91-410b-8fda-53c02e439893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc" Nov 03 15:27:53.530517 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:53.530484 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ef56ec-1f91-410b-8fda-53c02e439893-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f550b8c-85mjc\" (UID: \"09ef56ec-1f91-410b-8fda-53c02e439893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc" Nov 03 15:27:53.531524 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:53.531282 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ef56ec-1f91-410b-8fda-53c02e439893-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f550b8c-85mjc\" (UID: \"09ef56ec-1f91-410b-8fda-53c02e439893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc" Nov 03 15:27:53.544630 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:53.538985 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ef56ec-1f91-410b-8fda-53c02e439893-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f550b8c-85mjc\" (UID: \"09ef56ec-1f91-410b-8fda-53c02e439893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc" Nov 03 15:27:53.597647 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:53.593432 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb55f\" (UniqueName: \"kubernetes.io/projected/09ef56ec-1f91-410b-8fda-53c02e439893-kube-api-access-rb55f\") pod \"maas-api-revoke-sub-4f550b8c-85mjc\" (UID: \"09ef56ec-1f91-410b-8fda-53c02e439893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc" Nov 03 15:27:53.641178 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:53.640710 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc" Nov 03 15:27:54.048220 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:27:54.048188 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ef56ec_1f91_410b_8fda_53c02e439893.slice/crio-79d33fed9c6a50fb2a90fa7e36a89ed72beb4f6e9cf18313f7e21fd28993ee9f WatchSource:0}: Error finding container 79d33fed9c6a50fb2a90fa7e36a89ed72beb4f6e9cf18313f7e21fd28993ee9f: Status 404 returned error can't find the container with id 79d33fed9c6a50fb2a90fa7e36a89ed72beb4f6e9cf18313f7e21fd28993ee9f Nov 03 15:27:54.048332 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:54.048308 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc"] Nov 03 15:27:54.877702 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:54.877669 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc" event={"ID":"09ef56ec-1f91-410b-8fda-53c02e439893","Type":"ContainerStarted","Data":"1002bd399aefeb5aa7c2bf3d1b54a9829348cea27b763019e99921eb9e9f2645"} Nov 03 15:27:54.878081 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:54.877720 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc" event={"ID":"09ef56ec-1f91-410b-8fda-53c02e439893","Type":"ContainerStarted","Data":"79d33fed9c6a50fb2a90fa7e36a89ed72beb4f6e9cf18313f7e21fd28993ee9f"} Nov 03 15:27:54.997470 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:54.997418 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc" podStartSLOduration=1.997405146 podStartE2EDuration="1.997405146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:27:54.994831291 +0000 UTC m=+1184.837952834" watchObservedRunningTime="2026-11-03 15:27:54.997405146 +0000 UTC m=+1184.840526687" Nov 03 15:27:55.886883 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:55.886846 4557 generic.go:356] "Generic (PLEG): container finished" podID="09ef56ec-1f91-410b-8fda-53c02e439893" containerID="1002bd399aefeb5aa7c2bf3d1b54a9829348cea27b763019e99921eb9e9f2645" exitCode=0 Nov 03 15:27:55.887254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:55.886923 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc" event={"ID":"09ef56ec-1f91-410b-8fda-53c02e439893","Type":"ContainerDied","Data":"1002bd399aefeb5aa7c2bf3d1b54a9829348cea27b763019e99921eb9e9f2645"} Nov 03 15:27:57.026944 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:57.026918 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc" Nov 03 15:27:57.181185 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:57.181104 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ef56ec-1f91-410b-8fda-53c02e439893-kube-api-access-rb55f\" (UniqueName: \"kubernetes.io/projected/09ef56ec-1f91-410b-8fda-53c02e439893-kube-api-access-rb55f\") pod \"09ef56ec-1f91-410b-8fda-53c02e439893\" (UID: \"09ef56ec-1f91-410b-8fda-53c02e439893\") " Nov 03 15:27:57.181347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:57.181329 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ef56ec-1f91-410b-8fda-53c02e439893-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ef56ec-1f91-410b-8fda-53c02e439893-maas-api-xks-ca\") pod \"09ef56ec-1f91-410b-8fda-53c02e439893\" (UID: \"09ef56ec-1f91-410b-8fda-53c02e439893\") " Nov 03 15:27:57.181392 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:57.181365 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ef56ec-1f91-410b-8fda-53c02e439893-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ef56ec-1f91-410b-8fda-53c02e439893-maas-api-openshift-ca\") pod \"09ef56ec-1f91-410b-8fda-53c02e439893\" (UID: \"09ef56ec-1f91-410b-8fda-53c02e439893\") " Nov 03 15:27:57.181819 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:57.181793 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ef56ec-1f91-410b-8fda-53c02e439893-maas-api-openshift-ca" pod "09ef56ec-1f91-410b-8fda-53c02e439893" (UID: "09ef56ec-1f91-410b-8fda-53c02e439893"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:27:57.183347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:57.183321 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ef56ec-1f91-410b-8fda-53c02e439893-maas-api-xks-ca" pod "09ef56ec-1f91-410b-8fda-53c02e439893" (UID: "09ef56ec-1f91-410b-8fda-53c02e439893"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:27:57.184037 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:57.183963 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ef56ec-1f91-410b-8fda-53c02e439893-kube-api-access-rb55f" pod "09ef56ec-1f91-410b-8fda-53c02e439893" (UID: "09ef56ec-1f91-410b-8fda-53c02e439893"). InnerVolumeSpecName "kube-api-access-rb55f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:27:57.282764 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:57.282737 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb55f\" (UniqueName: \"kubernetes.io/projected/09ef56ec-1f91-410b-8fda-53c02e439893-kube-api-access-rb55f\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:27:57.282764 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:57.282762 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ef56ec-1f91-410b-8fda-53c02e439893-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:27:57.282921 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:57.282772 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ef56ec-1f91-410b-8fda-53c02e439893-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:27:57.900222 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:57.900194 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc" Nov 03 15:27:57.900395 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:57.900193 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc" event={"ID":"09ef56ec-1f91-410b-8fda-53c02e439893","Type":"ContainerDied","Data":"79d33fed9c6a50fb2a90fa7e36a89ed72beb4f6e9cf18313f7e21fd28993ee9f"} Nov 03 15:27:57.900395 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:57.900324 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79d33fed9c6a50fb2a90fa7e36a89ed72beb4f6e9cf18313f7e21fd28993ee9f" Nov 03 15:27:58.326103 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:58.326066 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:27:58.890651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:27:58.890612 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:28:01.921265 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:01.921234 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:28:03.240383 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:03.240289 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:28:05.475561 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:05.475521 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:28:06.062368 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:06.062333 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8"] Nov 03 15:28:06.112151 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:06.112115 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39928067-d6jz8"] Nov 03 15:28:06.287269 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:06.287240 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc614deb-ad31-4e6b-be3f-ddea755c0c7d" path="/var/lib/kubelet/pods/cc614deb-ad31-4e6b-be3f-ddea755c0c7d/volumes" Nov 03 15:28:07.994735 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:07.994704 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq"] Nov 03 15:28:07.995163 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:07.995151 4557 cpu_manager.go:414] "containerMap: removing container" podUID="09ef56ec-1f91-410b-8fda-53c02e439893" containerName="revoke-keys" Nov 03 15:28:07.995205 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:07.995164 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ef56ec-1f91-410b-8fda-53c02e439893" containerName="revoke-keys" Nov 03 15:28:07.995256 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:07.995247 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ef56ec-1f91-410b-8fda-53c02e439893" containerName="revoke-keys" Nov 03 15:28:07.998693 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:07.998677 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq" Nov 03 15:28:08.056600 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:08.056558 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq"] Nov 03 15:28:08.099068 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:08.099036 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7vts\" (UniqueName: \"kubernetes.io/projected/3df0dec4-7e3a-4e0d-8233-09336bcdc899-kube-api-access-x7vts\") pod \"maas-api-revoke-sub-63533408-rtqzq\" (UID: \"3df0dec4-7e3a-4e0d-8233-09336bcdc899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq" Nov 03 15:28:08.099199 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:08.099179 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df0dec4-7e3a-4e0d-8233-09336bcdc899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63533408-rtqzq\" (UID: \"3df0dec4-7e3a-4e0d-8233-09336bcdc899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq" Nov 03 15:28:08.099262 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:08.099246 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df0dec4-7e3a-4e0d-8233-09336bcdc899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63533408-rtqzq\" (UID: \"3df0dec4-7e3a-4e0d-8233-09336bcdc899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq" Nov 03 15:28:08.102105 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:08.102080 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:28:08.102105 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:08.102095 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:28:08.200422 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:08.200391 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df0dec4-7e3a-4e0d-8233-09336bcdc899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63533408-rtqzq\" (UID: \"3df0dec4-7e3a-4e0d-8233-09336bcdc899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq" Nov 03 15:28:08.200632 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:08.200434 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df0dec4-7e3a-4e0d-8233-09336bcdc899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63533408-rtqzq\" (UID: \"3df0dec4-7e3a-4e0d-8233-09336bcdc899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq" Nov 03 15:28:08.200632 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:08.200508 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7vts\" (UniqueName: \"kubernetes.io/projected/3df0dec4-7e3a-4e0d-8233-09336bcdc899-kube-api-access-x7vts\") pod \"maas-api-revoke-sub-63533408-rtqzq\" (UID: \"3df0dec4-7e3a-4e0d-8233-09336bcdc899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq" Nov 03 15:28:08.201176 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:08.201151 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df0dec4-7e3a-4e0d-8233-09336bcdc899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63533408-rtqzq\" (UID: \"3df0dec4-7e3a-4e0d-8233-09336bcdc899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq" Nov 03 15:28:08.202711 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:08.202694 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df0dec4-7e3a-4e0d-8233-09336bcdc899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63533408-rtqzq\" (UID: \"3df0dec4-7e3a-4e0d-8233-09336bcdc899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq" Nov 03 15:28:08.268425 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:08.268350 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7vts\" (UniqueName: \"kubernetes.io/projected/3df0dec4-7e3a-4e0d-8233-09336bcdc899-kube-api-access-x7vts\") pod \"maas-api-revoke-sub-63533408-rtqzq\" (UID: \"3df0dec4-7e3a-4e0d-8233-09336bcdc899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq" Nov 03 15:28:08.311440 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:08.311414 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq" Nov 03 15:28:08.356925 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:08.356881 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:28:08.688013 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:08.687982 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq"] Nov 03 15:28:08.688524 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:28:08.688501 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df0dec4_7e3a_4e0d_8233_09336bcdc899.slice/crio-9fc53cc05ae73425e0cc0cff534f98a97310dea79b44914bb4bbc2d0e0e9b8f5 WatchSource:0}: Error finding container 9fc53cc05ae73425e0cc0cff534f98a97310dea79b44914bb4bbc2d0e0e9b8f5: Status 404 returned error can't find the container with id 9fc53cc05ae73425e0cc0cff534f98a97310dea79b44914bb4bbc2d0e0e9b8f5 Nov 03 15:28:08.972237 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:08.972157 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq" event={"ID":"3df0dec4-7e3a-4e0d-8233-09336bcdc899","Type":"ContainerStarted","Data":"9fc53cc05ae73425e0cc0cff534f98a97310dea79b44914bb4bbc2d0e0e9b8f5"} Nov 03 15:28:09.980036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:09.979992 4557 generic.go:356] "Generic (PLEG): container finished" podID="3df0dec4-7e3a-4e0d-8233-09336bcdc899" containerID="b6838e8c3ac2a0c614b49cbfe3373f14d2dd2d3744c6b288486dbfb17f8c5554" exitCode=0 Nov 03 15:28:09.980448 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:09.980074 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq" event={"ID":"3df0dec4-7e3a-4e0d-8233-09336bcdc899","Type":"ContainerDied","Data":"b6838e8c3ac2a0c614b49cbfe3373f14d2dd2d3744c6b288486dbfb17f8c5554"} Nov 03 15:28:10.403628 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:10.403569 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:28:10.405430 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:10.405402 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:28:11.138135 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:11.138104 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq" Nov 03 15:28:11.233876 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:11.233844 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3df0dec4-7e3a-4e0d-8233-09336bcdc899-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df0dec4-7e3a-4e0d-8233-09336bcdc899-maas-api-openshift-ca\") pod \"3df0dec4-7e3a-4e0d-8233-09336bcdc899\" (UID: \"3df0dec4-7e3a-4e0d-8233-09336bcdc899\") " Nov 03 15:28:11.234088 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:11.234029 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3df0dec4-7e3a-4e0d-8233-09336bcdc899-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df0dec4-7e3a-4e0d-8233-09336bcdc899-maas-api-xks-ca\") pod \"3df0dec4-7e3a-4e0d-8233-09336bcdc899\" (UID: \"3df0dec4-7e3a-4e0d-8233-09336bcdc899\") " Nov 03 15:28:11.234088 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:11.234073 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df0dec4-7e3a-4e0d-8233-09336bcdc899-kube-api-access-x7vts\" (UniqueName: \"kubernetes.io/projected/3df0dec4-7e3a-4e0d-8233-09336bcdc899-kube-api-access-x7vts\") pod \"3df0dec4-7e3a-4e0d-8233-09336bcdc899\" (UID: \"3df0dec4-7e3a-4e0d-8233-09336bcdc899\") " Nov 03 15:28:11.234295 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:11.234264 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df0dec4-7e3a-4e0d-8233-09336bcdc899-maas-api-openshift-ca" pod "3df0dec4-7e3a-4e0d-8233-09336bcdc899" (UID: "3df0dec4-7e3a-4e0d-8233-09336bcdc899"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:28:11.234480 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:11.234463 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df0dec4-7e3a-4e0d-8233-09336bcdc899-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:28:11.236172 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:11.236147 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3df0dec4-7e3a-4e0d-8233-09336bcdc899-maas-api-xks-ca" pod "3df0dec4-7e3a-4e0d-8233-09336bcdc899" (UID: "3df0dec4-7e3a-4e0d-8233-09336bcdc899"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:28:11.236267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:11.236254 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df0dec4-7e3a-4e0d-8233-09336bcdc899-kube-api-access-x7vts" pod "3df0dec4-7e3a-4e0d-8233-09336bcdc899" (UID: "3df0dec4-7e3a-4e0d-8233-09336bcdc899"). InnerVolumeSpecName "kube-api-access-x7vts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:28:11.335900 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:11.335861 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df0dec4-7e3a-4e0d-8233-09336bcdc899-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:28:11.335900 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:11.335895 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7vts\" (UniqueName: \"kubernetes.io/projected/3df0dec4-7e3a-4e0d-8233-09336bcdc899-kube-api-access-x7vts\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:28:11.993972 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:11.993941 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq" Nov 03 15:28:11.994151 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:11.993980 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq" event={"ID":"3df0dec4-7e3a-4e0d-8233-09336bcdc899","Type":"ContainerDied","Data":"9fc53cc05ae73425e0cc0cff534f98a97310dea79b44914bb4bbc2d0e0e9b8f5"} Nov 03 15:28:11.994151 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:11.994036 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fc53cc05ae73425e0cc0cff534f98a97310dea79b44914bb4bbc2d0e0e9b8f5" Nov 03 15:28:13.264341 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:13.264296 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:28:16.017572 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.017538 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm"] Nov 03 15:28:16.018042 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.018028 4557 cpu_manager.go:414] "containerMap: removing container" podUID="3df0dec4-7e3a-4e0d-8233-09336bcdc899" containerName="revoke-keys" Nov 03 15:28:16.018082 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.018043 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3df0dec4-7e3a-4e0d-8233-09336bcdc899" containerName="revoke-keys" Nov 03 15:28:16.018139 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.018130 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="3df0dec4-7e3a-4e0d-8233-09336bcdc899" containerName="revoke-keys" Nov 03 15:28:16.021485 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.021470 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.086487 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.086451 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.086682 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.086522 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.086682 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.086653 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.086799 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.086708 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.086835 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.086801 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.086835 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.086819 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7snj7\" (UniqueName: \"kubernetes.io/projected/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-kube-api-access-7snj7\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.086957 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.086936 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.086993 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.086965 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.087039 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.087027 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.187832 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.187793 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.188007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.187945 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.188095 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.188004 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.188095 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.188030 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7snj7\" (UniqueName: \"kubernetes.io/projected/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-kube-api-access-7snj7\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.188227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.188118 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.188227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.188165 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.188343 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.188241 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.188343 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.188252 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.188343 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.188279 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.188500 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.188362 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.188557 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.188496 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.188715 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.188687 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.188969 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.188941 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.189317 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.189291 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.190852 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.190821 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.191513 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.191491 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.234861 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.234834 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm"] Nov 03 15:28:16.250033 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.250008 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.250146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.250113 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7snj7\" (UniqueName: \"kubernetes.io/projected/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-kube-api-access-7snj7\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.335910 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.335885 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:16.530421 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:28:16.530386 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:28:16.769917 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.769892 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm"] Nov 03 15:28:16.771150 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:28:16.771102 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d60ef8_6b7d_45af_8b6b_5579fb6ce960.slice/crio-553771b10cc76e4d78d7e04d8726ba4413c95d3289e110c11ef3a8e0b64b794c WatchSource:0}: Error finding container 553771b10cc76e4d78d7e04d8726ba4413c95d3289e110c11ef3a8e0b64b794c: Status 404 returned error can't find the container with id 553771b10cc76e4d78d7e04d8726ba4413c95d3289e110c11ef3a8e0b64b794c Nov 03 15:28:16.773625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.773598 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:28:16.773713 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.773655 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:28:16.773713 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:16.773684 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:28:17.027874 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:17.027741 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" event={"ID":"93d60ef8-6b7d-45af-8b6b-5579fb6ce960","Type":"ContainerStarted","Data":"8df1bf69ec9ad974fe9f2269a137969dd02ccd9f948bd3d6dbbffc5e30d826dc"} Nov 03 15:28:17.027874 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:17.027805 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" event={"ID":"93d60ef8-6b7d-45af-8b6b-5579fb6ce960","Type":"ContainerStarted","Data":"553771b10cc76e4d78d7e04d8726ba4413c95d3289e110c11ef3a8e0b64b794c"} Nov 03 15:28:17.157148 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:17.157095 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" podStartSLOduration=2.157082615 podStartE2EDuration="2.157082615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:28:17.15414722 +0000 UTC m=+1206.997268763" watchObservedRunningTime="2026-11-03 15:28:17.157082615 +0000 UTC m=+1207.000204156" Nov 03 15:28:17.336384 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:17.336347 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:17.337867 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:17.337843 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 15:28:17.338017 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:17.337888 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" podUID="93d60ef8-6b7d-45af-8b6b-5579fb6ce960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 15:28:18.151233 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:18.151190 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:28:18.336664 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:18.336629 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 15:28:18.336833 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:18.336690 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" podUID="93d60ef8-6b7d-45af-8b6b-5579fb6ce960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 15:28:19.337270 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:19.337228 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 15:28:19.337683 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:19.337288 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" podUID="93d60ef8-6b7d-45af-8b6b-5579fb6ce960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 15:28:20.336849 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:20.336811 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 15:28:20.337018 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:20.336871 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" podUID="93d60ef8-6b7d-45af-8b6b-5579fb6ce960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 15:28:21.336571 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:21.336536 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 15:28:21.337120 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:21.336623 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" podUID="93d60ef8-6b7d-45af-8b6b-5579fb6ce960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 15:28:22.336964 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:22.336927 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 15:28:22.337421 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:22.336986 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" podUID="93d60ef8-6b7d-45af-8b6b-5579fb6ce960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 15:28:23.205849 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:23.205637 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:28:23.336378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:23.336344 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 15:28:23.336574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:23.336409 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" podUID="93d60ef8-6b7d-45af-8b6b-5579fb6ce960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 15:28:24.363472 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:24.363437 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:24.364080 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:24.364046 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:24.364837 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:24.364811 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:28:24.761682 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:24.761605 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb"] Nov 03 15:28:24.765631 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:24.765615 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb" Nov 03 15:28:24.821023 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:24.820995 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:28:24.821190 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:24.821124 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:28:24.825913 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:24.825885 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb"] Nov 03 15:28:24.895378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:24.895344 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28bef2c6-f2d1-4087-8089-4e15c7d51db2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7736cce7-fhdbb\" (UID: \"28bef2c6-f2d1-4087-8089-4e15c7d51db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb" Nov 03 15:28:24.895558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:24.895534 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28bef2c6-f2d1-4087-8089-4e15c7d51db2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7736cce7-fhdbb\" (UID: \"28bef2c6-f2d1-4087-8089-4e15c7d51db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb" Nov 03 15:28:24.895679 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:24.895660 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxbp9\" (UniqueName: \"kubernetes.io/projected/28bef2c6-f2d1-4087-8089-4e15c7d51db2-kube-api-access-lxbp9\") pod \"maas-api-revoke-sub-7736cce7-fhdbb\" (UID: \"28bef2c6-f2d1-4087-8089-4e15c7d51db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb" Nov 03 15:28:24.996985 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:24.996948 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28bef2c6-f2d1-4087-8089-4e15c7d51db2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7736cce7-fhdbb\" (UID: \"28bef2c6-f2d1-4087-8089-4e15c7d51db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb" Nov 03 15:28:24.997165 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:24.997051 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28bef2c6-f2d1-4087-8089-4e15c7d51db2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7736cce7-fhdbb\" (UID: \"28bef2c6-f2d1-4087-8089-4e15c7d51db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb" Nov 03 15:28:24.997165 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:24.997085 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxbp9\" (UniqueName: \"kubernetes.io/projected/28bef2c6-f2d1-4087-8089-4e15c7d51db2-kube-api-access-lxbp9\") pod \"maas-api-revoke-sub-7736cce7-fhdbb\" (UID: \"28bef2c6-f2d1-4087-8089-4e15c7d51db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb" Nov 03 15:28:24.997703 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:24.997676 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28bef2c6-f2d1-4087-8089-4e15c7d51db2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7736cce7-fhdbb\" (UID: \"28bef2c6-f2d1-4087-8089-4e15c7d51db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb" Nov 03 15:28:24.999281 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:24.999262 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28bef2c6-f2d1-4087-8089-4e15c7d51db2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7736cce7-fhdbb\" (UID: \"28bef2c6-f2d1-4087-8089-4e15c7d51db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb" Nov 03 15:28:25.059304 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:25.059270 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxbp9\" (UniqueName: \"kubernetes.io/projected/28bef2c6-f2d1-4087-8089-4e15c7d51db2-kube-api-access-lxbp9\") pod \"maas-api-revoke-sub-7736cce7-fhdbb\" (UID: \"28bef2c6-f2d1-4087-8089-4e15c7d51db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb" Nov 03 15:28:25.079215 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:25.079184 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb" Nov 03 15:28:25.458870 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:25.458838 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb"] Nov 03 15:28:25.459676 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:28:25.459645 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28bef2c6_f2d1_4087_8089_4e15c7d51db2.slice/crio-9119197708d1513d928bfcd0a7db3db695258c71d21774f24d64b606027a4924 WatchSource:0}: Error finding container 9119197708d1513d928bfcd0a7db3db695258c71d21774f24d64b606027a4924: Status 404 returned error can't find the container with id 9119197708d1513d928bfcd0a7db3db695258c71d21774f24d64b606027a4924 Nov 03 15:28:26.087779 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:26.087741 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb" event={"ID":"28bef2c6-f2d1-4087-8089-4e15c7d51db2","Type":"ContainerStarted","Data":"0eabd28494280a4207f7e9260052cf2ad254edee05a062d0502cc8a34ecdb2ea"} Nov 03 15:28:26.087956 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:26.087799 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb" event={"ID":"28bef2c6-f2d1-4087-8089-4e15c7d51db2","Type":"ContainerStarted","Data":"9119197708d1513d928bfcd0a7db3db695258c71d21774f24d64b606027a4924"} Nov 03 15:28:26.208574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:26.208521 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb" podStartSLOduration=2.208506735 podStartE2EDuration="2.208506735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:28:26.204614712 +0000 UTC m=+1216.047736252" watchObservedRunningTime="2026-11-03 15:28:26.208506735 +0000 UTC m=+1216.051628320" Nov 03 15:28:27.095800 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:27.095767 4557 generic.go:356] "Generic (PLEG): container finished" podID="28bef2c6-f2d1-4087-8089-4e15c7d51db2" containerID="0eabd28494280a4207f7e9260052cf2ad254edee05a062d0502cc8a34ecdb2ea" exitCode=0 Nov 03 15:28:27.096204 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:27.095805 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb" event={"ID":"28bef2c6-f2d1-4087-8089-4e15c7d51db2","Type":"ContainerDied","Data":"0eabd28494280a4207f7e9260052cf2ad254edee05a062d0502cc8a34ecdb2ea"} Nov 03 15:28:28.087042 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:28.087006 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:28:28.259543 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:28.259520 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb" Nov 03 15:28:28.333570 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:28.333541 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28bef2c6-f2d1-4087-8089-4e15c7d51db2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28bef2c6-f2d1-4087-8089-4e15c7d51db2-maas-api-xks-ca\") pod \"28bef2c6-f2d1-4087-8089-4e15c7d51db2\" (UID: \"28bef2c6-f2d1-4087-8089-4e15c7d51db2\") " Nov 03 15:28:28.333769 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:28.333755 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28bef2c6-f2d1-4087-8089-4e15c7d51db2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28bef2c6-f2d1-4087-8089-4e15c7d51db2-maas-api-openshift-ca\") pod \"28bef2c6-f2d1-4087-8089-4e15c7d51db2\" (UID: \"28bef2c6-f2d1-4087-8089-4e15c7d51db2\") " Nov 03 15:28:28.333817 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:28.333801 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28bef2c6-f2d1-4087-8089-4e15c7d51db2-kube-api-access-lxbp9\" (UniqueName: \"kubernetes.io/projected/28bef2c6-f2d1-4087-8089-4e15c7d51db2-kube-api-access-lxbp9\") pod \"28bef2c6-f2d1-4087-8089-4e15c7d51db2\" (UID: \"28bef2c6-f2d1-4087-8089-4e15c7d51db2\") " Nov 03 15:28:28.334156 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:28.334132 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28bef2c6-f2d1-4087-8089-4e15c7d51db2-maas-api-openshift-ca" pod "28bef2c6-f2d1-4087-8089-4e15c7d51db2" (UID: "28bef2c6-f2d1-4087-8089-4e15c7d51db2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:28:28.334247 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:28.334231 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28bef2c6-f2d1-4087-8089-4e15c7d51db2-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:28:28.335466 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:28.335442 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28bef2c6-f2d1-4087-8089-4e15c7d51db2-maas-api-xks-ca" pod "28bef2c6-f2d1-4087-8089-4e15c7d51db2" (UID: "28bef2c6-f2d1-4087-8089-4e15c7d51db2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:28:28.335772 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:28.335751 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28bef2c6-f2d1-4087-8089-4e15c7d51db2-kube-api-access-lxbp9" pod "28bef2c6-f2d1-4087-8089-4e15c7d51db2" (UID: "28bef2c6-f2d1-4087-8089-4e15c7d51db2"). InnerVolumeSpecName "kube-api-access-lxbp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:28:28.435362 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:28.435333 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxbp9\" (UniqueName: \"kubernetes.io/projected/28bef2c6-f2d1-4087-8089-4e15c7d51db2-kube-api-access-lxbp9\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:28:28.435362 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:28.435360 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28bef2c6-f2d1-4087-8089-4e15c7d51db2-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:28:29.115537 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:29.115506 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb" Nov 03 15:28:29.115731 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:29.115543 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb" event={"ID":"28bef2c6-f2d1-4087-8089-4e15c7d51db2","Type":"ContainerDied","Data":"9119197708d1513d928bfcd0a7db3db695258c71d21774f24d64b606027a4924"} Nov 03 15:28:29.115731 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:29.115624 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9119197708d1513d928bfcd0a7db3db695258c71d21774f24d64b606027a4924" Nov 03 15:28:33.032694 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.032657 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:28:33.055884 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.055854 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8"] Nov 03 15:28:33.102896 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.102862 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a24ee7-pnbp8"] Nov 03 15:28:33.813283 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.813248 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v"] Nov 03 15:28:33.813766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.813752 4557 cpu_manager.go:414] "containerMap: removing container" podUID="28bef2c6-f2d1-4087-8089-4e15c7d51db2" containerName="revoke-keys" Nov 03 15:28:33.813826 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.813767 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28bef2c6-f2d1-4087-8089-4e15c7d51db2" containerName="revoke-keys" Nov 03 15:28:33.813866 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.813852 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="28bef2c6-f2d1-4087-8089-4e15c7d51db2" containerName="revoke-keys" Nov 03 15:28:33.817328 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.817313 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:33.876651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.876617 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v"] Nov 03 15:28:33.897317 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.897292 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-home\") pod \"distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:33.897469 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.897371 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggrpb\" (UniqueName: \"kubernetes.io/projected/9c968070-2d65-4501-b504-e96664a97f8d-kube-api-access-ggrpb\") pod \"distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:33.897469 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.897413 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-model-cache\") pod \"distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:33.897634 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.897561 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-tmp-dir\") pod \"distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:33.897634 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.897619 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-dshm\") pod \"distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:33.897714 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.897652 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c968070-2d65-4501-b504-e96664a97f8d-tls-certs\") pod \"distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:33.921969 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.921933 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bc1f07\"/\"distinct-2-w4-bc1f07-kserve-self-signed-certs\"" Nov 03 15:28:33.962096 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.962047 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:28:33.999120 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.999079 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-model-cache\") pod \"distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:33.999327 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.999230 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-tmp-dir\") pod \"distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:33.999327 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.999283 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-dshm\") pod \"distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:33.999327 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.999313 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c968070-2d65-4501-b504-e96664a97f8d-tls-certs\") pod \"distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:33.999496 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.999353 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-home\") pod \"distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:33.999496 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.999455 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggrpb\" (UniqueName: \"kubernetes.io/projected/9c968070-2d65-4501-b504-e96664a97f8d-kube-api-access-ggrpb\") pod \"distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:33.999756 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.999547 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-model-cache\") pod \"distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:33.999957 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.999820 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-home\") pod \"distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:33.999957 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:33.999939 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-tmp-dir\") pod \"distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:34.002228 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:34.002203 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c968070-2d65-4501-b504-e96664a97f8d-tls-certs\") pod \"distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:34.002357 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:34.002294 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-dshm\") pod \"distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:34.064425 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:34.064347 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggrpb\" (UniqueName: \"kubernetes.io/projected/9c968070-2d65-4501-b504-e96664a97f8d-kube-api-access-ggrpb\") pod \"distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:34.130188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:34.130149 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:34.291355 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:34.291324 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a8b5f8d-947f-43e9-8603-05a8b6cb4930" path="/var/lib/kubelet/pods/8a8b5f8d-947f-43e9-8603-05a8b6cb4930/volumes" Nov 03 15:28:34.517697 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:34.517666 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v"] Nov 03 15:28:34.517917 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:28:34.517891 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c968070_2d65_4501_b504_e96664a97f8d.slice/crio-0f4e6609dc4c2706a8e44631433c18beff315e92fe4612c4add8a228941f5945 WatchSource:0}: Error finding container 0f4e6609dc4c2706a8e44631433c18beff315e92fe4612c4add8a228941f5945: Status 404 returned error can't find the container with id 0f4e6609dc4c2706a8e44631433c18beff315e92fe4612c4add8a228941f5945 Nov 03 15:28:34.773357 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:34.773282 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng"] Nov 03 15:28:34.777112 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:34.777093 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng" Nov 03 15:28:34.855697 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:34.855667 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng"] Nov 03 15:28:34.862516 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:34.862490 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:28:34.862516 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:34.862487 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:28:34.910990 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:34.910959 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7rnt\" (UniqueName: \"kubernetes.io/projected/f7fceb3c-3f88-4f23-9387-099c4287f353-kube-api-access-h7rnt\") pod \"maas-api-revoke-sub-0c0addb6-hwbng\" (UID: \"f7fceb3c-3f88-4f23-9387-099c4287f353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng" Nov 03 15:28:34.911147 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:34.911007 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fceb3c-3f88-4f23-9387-099c4287f353-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c0addb6-hwbng\" (UID: \"f7fceb3c-3f88-4f23-9387-099c4287f353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng" Nov 03 15:28:34.911147 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:34.911111 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fceb3c-3f88-4f23-9387-099c4287f353-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c0addb6-hwbng\" (UID: \"f7fceb3c-3f88-4f23-9387-099c4287f353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng" Nov 03 15:28:35.012830 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:35.012797 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7rnt\" (UniqueName: \"kubernetes.io/projected/f7fceb3c-3f88-4f23-9387-099c4287f353-kube-api-access-h7rnt\") pod \"maas-api-revoke-sub-0c0addb6-hwbng\" (UID: \"f7fceb3c-3f88-4f23-9387-099c4287f353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng" Nov 03 15:28:35.012959 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:35.012838 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fceb3c-3f88-4f23-9387-099c4287f353-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c0addb6-hwbng\" (UID: \"f7fceb3c-3f88-4f23-9387-099c4287f353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng" Nov 03 15:28:35.012959 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:35.012866 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fceb3c-3f88-4f23-9387-099c4287f353-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c0addb6-hwbng\" (UID: \"f7fceb3c-3f88-4f23-9387-099c4287f353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng" Nov 03 15:28:35.013439 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:35.013414 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fceb3c-3f88-4f23-9387-099c4287f353-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c0addb6-hwbng\" (UID: \"f7fceb3c-3f88-4f23-9387-099c4287f353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng" Nov 03 15:28:35.015123 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:35.015099 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fceb3c-3f88-4f23-9387-099c4287f353-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c0addb6-hwbng\" (UID: \"f7fceb3c-3f88-4f23-9387-099c4287f353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng" Nov 03 15:28:35.075538 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:35.075500 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7rnt\" (UniqueName: \"kubernetes.io/projected/f7fceb3c-3f88-4f23-9387-099c4287f353-kube-api-access-h7rnt\") pod \"maas-api-revoke-sub-0c0addb6-hwbng\" (UID: \"f7fceb3c-3f88-4f23-9387-099c4287f353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng" Nov 03 15:28:35.090056 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:35.090035 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng" Nov 03 15:28:35.161766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:35.161731 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" event={"ID":"9c968070-2d65-4501-b504-e96664a97f8d","Type":"ContainerStarted","Data":"8cc513c36f8d563661286f7d6492344d3dfa63e16ced3ab23979e9107ffe1500"} Nov 03 15:28:35.161905 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:35.161792 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" event={"ID":"9c968070-2d65-4501-b504-e96664a97f8d","Type":"ContainerStarted","Data":"0f4e6609dc4c2706a8e44631433c18beff315e92fe4612c4add8a228941f5945"} Nov 03 15:28:35.287430 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:35.287378 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podStartSLOduration=2.2873636299999998 podStartE2EDuration="2.28736363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:28:35.284648999 +0000 UTC m=+1225.127770542" watchObservedRunningTime="2026-11-03 15:28:35.28736363 +0000 UTC m=+1225.130485172" Nov 03 15:28:35.468864 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:28:35.468827 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fceb3c_3f88_4f23_9387_099c4287f353.slice/crio-efd2d4a03e4710c5f02e1aa0d802c4a47c34e840df49bd5d970bfabcd91a0403 WatchSource:0}: Error finding container efd2d4a03e4710c5f02e1aa0d802c4a47c34e840df49bd5d970bfabcd91a0403: Status 404 returned error can't find the container with id efd2d4a03e4710c5f02e1aa0d802c4a47c34e840df49bd5d970bfabcd91a0403 Nov 03 15:28:35.468864 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:35.468834 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng"] Nov 03 15:28:36.130392 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:36.130356 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:36.169745 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:36.169710 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng" event={"ID":"f7fceb3c-3f88-4f23-9387-099c4287f353","Type":"ContainerStarted","Data":"adfbf2d27191a42b2690a305edf9154e20cf88fce10fa4bcc43b80533856dd19"} Nov 03 15:28:36.169908 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:36.169791 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng" event={"ID":"f7fceb3c-3f88-4f23-9387-099c4287f353","Type":"ContainerStarted","Data":"efd2d4a03e4710c5f02e1aa0d802c4a47c34e840df49bd5d970bfabcd91a0403"} Nov 03 15:28:36.289633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:36.289563 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng" podStartSLOduration=2.28955026 podStartE2EDuration="2.28955026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:28:36.285822719 +0000 UTC m=+1226.128944407" watchObservedRunningTime="2026-11-03 15:28:36.28955026 +0000 UTC m=+1226.132671818" Nov 03 15:28:37.177470 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:37.177434 4557 generic.go:356] "Generic (PLEG): container finished" podID="f7fceb3c-3f88-4f23-9387-099c4287f353" containerID="adfbf2d27191a42b2690a305edf9154e20cf88fce10fa4bcc43b80533856dd19" exitCode=0 Nov 03 15:28:37.177947 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:37.177518 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng" event={"ID":"f7fceb3c-3f88-4f23-9387-099c4287f353","Type":"ContainerDied","Data":"adfbf2d27191a42b2690a305edf9154e20cf88fce10fa4bcc43b80533856dd19"} Nov 03 15:28:38.050861 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:38.050827 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4"] Nov 03 15:28:38.077640 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:38.077607 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b726b97-8pmb4"] Nov 03 15:28:38.127335 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:38.127307 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:28:38.288662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:38.288618 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47a98824-02da-44d2-95d6-971e5fb4f3a2" path="/var/lib/kubelet/pods/47a98824-02da-44d2-95d6-971e5fb4f3a2/volumes" Nov 03 15:28:38.335163 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:38.335142 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng" Nov 03 15:28:38.455365 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:38.455334 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7fceb3c-3f88-4f23-9387-099c4287f353-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fceb3c-3f88-4f23-9387-099c4287f353-maas-api-xks-ca\") pod \"f7fceb3c-3f88-4f23-9387-099c4287f353\" (UID: \"f7fceb3c-3f88-4f23-9387-099c4287f353\") " Nov 03 15:28:38.455524 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:38.455489 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7fceb3c-3f88-4f23-9387-099c4287f353-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fceb3c-3f88-4f23-9387-099c4287f353-maas-api-openshift-ca\") pod \"f7fceb3c-3f88-4f23-9387-099c4287f353\" (UID: \"f7fceb3c-3f88-4f23-9387-099c4287f353\") " Nov 03 15:28:38.455713 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:38.455693 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7fceb3c-3f88-4f23-9387-099c4287f353-kube-api-access-h7rnt\" (UniqueName: \"kubernetes.io/projected/f7fceb3c-3f88-4f23-9387-099c4287f353-kube-api-access-h7rnt\") pod \"f7fceb3c-3f88-4f23-9387-099c4287f353\" (UID: \"f7fceb3c-3f88-4f23-9387-099c4287f353\") " Nov 03 15:28:38.455854 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:38.455825 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7fceb3c-3f88-4f23-9387-099c4287f353-maas-api-openshift-ca" pod "f7fceb3c-3f88-4f23-9387-099c4287f353" (UID: "f7fceb3c-3f88-4f23-9387-099c4287f353"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:28:38.456135 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:38.456120 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fceb3c-3f88-4f23-9387-099c4287f353-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:28:38.457511 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:38.457483 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7fceb3c-3f88-4f23-9387-099c4287f353-maas-api-xks-ca" pod "f7fceb3c-3f88-4f23-9387-099c4287f353" (UID: "f7fceb3c-3f88-4f23-9387-099c4287f353"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:28:38.457909 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:38.457886 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7fceb3c-3f88-4f23-9387-099c4287f353-kube-api-access-h7rnt" pod "f7fceb3c-3f88-4f23-9387-099c4287f353" (UID: "f7fceb3c-3f88-4f23-9387-099c4287f353"). InnerVolumeSpecName "kube-api-access-h7rnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:28:38.557531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:38.557487 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fceb3c-3f88-4f23-9387-099c4287f353-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:28:38.557531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:38.557528 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7rnt\" (UniqueName: \"kubernetes.io/projected/f7fceb3c-3f88-4f23-9387-099c4287f353-kube-api-access-h7rnt\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:28:39.193808 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:39.193774 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng" event={"ID":"f7fceb3c-3f88-4f23-9387-099c4287f353","Type":"ContainerDied","Data":"efd2d4a03e4710c5f02e1aa0d802c4a47c34e840df49bd5d970bfabcd91a0403"} Nov 03 15:28:39.193963 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:39.193829 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efd2d4a03e4710c5f02e1aa0d802c4a47c34e840df49bd5d970bfabcd91a0403" Nov 03 15:28:39.193963 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:39.193780 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng" Nov 03 15:28:40.579175 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:40.579143 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt"] Nov 03 15:28:40.579851 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:40.579825 4557 cpu_manager.go:414] "containerMap: removing container" podUID="f7fceb3c-3f88-4f23-9387-099c4287f353" containerName="revoke-keys" Nov 03 15:28:40.579963 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:40.579852 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7fceb3c-3f88-4f23-9387-099c4287f353" containerName="revoke-keys" Nov 03 15:28:40.580023 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:40.580002 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7fceb3c-3f88-4f23-9387-099c4287f353" containerName="revoke-keys" Nov 03 15:28:40.583650 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:40.583630 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt" Nov 03 15:28:40.679574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:40.679543 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa234c5e-brglt\" (UID: \"b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt" Nov 03 15:28:40.679740 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:40.679665 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmkb\" (UniqueName: \"kubernetes.io/projected/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-kube-api-access-6hmkb\") pod \"maas-api-revoke-sub-fa234c5e-brglt\" (UID: \"b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt" Nov 03 15:28:40.679740 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:40.679719 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa234c5e-brglt\" (UID: \"b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt" Nov 03 15:28:40.685949 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:40.685926 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:28:40.686064 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:40.685974 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:28:40.689684 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:40.689666 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt"] Nov 03 15:28:40.781167 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:40.781135 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa234c5e-brglt\" (UID: \"b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt" Nov 03 15:28:40.781304 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:40.781209 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmkb\" (UniqueName: \"kubernetes.io/projected/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-kube-api-access-6hmkb\") pod \"maas-api-revoke-sub-fa234c5e-brglt\" (UID: \"b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt" Nov 03 15:28:40.781304 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:40.781244 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa234c5e-brglt\" (UID: \"b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt" Nov 03 15:28:40.781917 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:40.781897 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa234c5e-brglt\" (UID: \"b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt" Nov 03 15:28:40.783497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:40.783473 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa234c5e-brglt\" (UID: \"b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt" Nov 03 15:28:40.842945 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:40.842881 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmkb\" (UniqueName: \"kubernetes.io/projected/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-kube-api-access-6hmkb\") pod \"maas-api-revoke-sub-fa234c5e-brglt\" (UID: \"b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt" Nov 03 15:28:40.898064 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:40.898012 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt" Nov 03 15:28:41.283393 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:41.283365 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt"] Nov 03 15:28:41.284130 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:28:41.284109 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ba5d83_546e_4eaa_b14e_7ffd7abfa3b0.slice/crio-9840aaaf92d39de805aacddead9abbd2faaaed416aad18c79b0ed45b2b58ddbc WatchSource:0}: Error finding container 9840aaaf92d39de805aacddead9abbd2faaaed416aad18c79b0ed45b2b58ddbc: Status 404 returned error can't find the container with id 9840aaaf92d39de805aacddead9abbd2faaaed416aad18c79b0ed45b2b58ddbc Nov 03 15:28:42.214471 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:42.214438 4557 generic.go:356] "Generic (PLEG): container finished" podID="b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0" containerID="a74f535326a2a9dfb2e7d00d1090274de9e02ac21b7433b1eb8ced37d4bc65a6" exitCode=0 Nov 03 15:28:42.214888 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:42.214519 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt" event={"ID":"b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0","Type":"ContainerDied","Data":"a74f535326a2a9dfb2e7d00d1090274de9e02ac21b7433b1eb8ced37d4bc65a6"} Nov 03 15:28:42.214888 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:42.214600 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt" event={"ID":"b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0","Type":"ContainerStarted","Data":"9840aaaf92d39de805aacddead9abbd2faaaed416aad18c79b0ed45b2b58ddbc"} Nov 03 15:28:43.115083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:43.115040 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:28:43.379893 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:43.379826 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt" Nov 03 15:28:43.513097 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:43.513065 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-kube-api-access-6hmkb\" (UniqueName: \"kubernetes.io/projected/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-kube-api-access-6hmkb\") pod \"b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0\" (UID: \"b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0\") " Nov 03 15:28:43.513264 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:43.513193 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-maas-api-xks-ca\") pod \"b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0\" (UID: \"b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0\") " Nov 03 15:28:43.513354 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:43.513280 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-maas-api-openshift-ca\") pod \"b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0\" (UID: \"b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0\") " Nov 03 15:28:43.513717 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:43.513688 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-maas-api-openshift-ca" pod "b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0" (UID: "b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:28:43.513865 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:43.513786 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:28:43.515312 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:43.515286 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-kube-api-access-6hmkb" pod "b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0" (UID: "b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0"). InnerVolumeSpecName "kube-api-access-6hmkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:28:43.515556 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:43.515540 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-maas-api-xks-ca" pod "b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0" (UID: "b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:28:43.614796 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:43.614767 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hmkb\" (UniqueName: \"kubernetes.io/projected/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-kube-api-access-6hmkb\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:28:43.614796 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:43.614795 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:28:44.130847 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:44.130811 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:44.143845 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:44.143811 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:44.228195 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:44.228171 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt" Nov 03 15:28:44.228349 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:44.228210 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt" event={"ID":"b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0","Type":"ContainerDied","Data":"9840aaaf92d39de805aacddead9abbd2faaaed416aad18c79b0ed45b2b58ddbc"} Nov 03 15:28:44.228349 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:44.228258 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9840aaaf92d39de805aacddead9abbd2faaaed416aad18c79b0ed45b2b58ddbc" Nov 03 15:28:44.242554 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:44.242533 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:28:46.530868 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:28:46.530835 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:28:47.823080 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:47.823047 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp"] Nov 03 15:28:47.823536 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:47.823523 4557 cpu_manager.go:414] "containerMap: removing container" podUID="b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0" containerName="revoke-keys" Nov 03 15:28:47.823604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:47.823537 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0" containerName="revoke-keys" Nov 03 15:28:47.823653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:47.823643 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0" containerName="revoke-keys" Nov 03 15:28:47.826851 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:47.826835 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp" Nov 03 15:28:47.881756 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:47.881719 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca2f9ef-6560-4b12-a003-1980889cec73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db5e2ad8-kqdnp\" (UID: \"eca2f9ef-6560-4b12-a003-1980889cec73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp" Nov 03 15:28:47.881905 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:47.881830 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca2f9ef-6560-4b12-a003-1980889cec73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db5e2ad8-kqdnp\" (UID: \"eca2f9ef-6560-4b12-a003-1980889cec73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp" Nov 03 15:28:47.882062 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:47.882038 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl826\" (UniqueName: \"kubernetes.io/projected/eca2f9ef-6560-4b12-a003-1980889cec73-kube-api-access-bl826\") pod \"maas-api-revoke-sub-db5e2ad8-kqdnp\" (UID: \"eca2f9ef-6560-4b12-a003-1980889cec73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp" Nov 03 15:28:47.885932 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:47.885909 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp"] Nov 03 15:28:47.931241 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:47.931214 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:28:47.931241 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:47.931238 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:28:47.982835 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:47.982807 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl826\" (UniqueName: \"kubernetes.io/projected/eca2f9ef-6560-4b12-a003-1980889cec73-kube-api-access-bl826\") pod \"maas-api-revoke-sub-db5e2ad8-kqdnp\" (UID: \"eca2f9ef-6560-4b12-a003-1980889cec73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp" Nov 03 15:28:47.982983 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:47.982856 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca2f9ef-6560-4b12-a003-1980889cec73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db5e2ad8-kqdnp\" (UID: \"eca2f9ef-6560-4b12-a003-1980889cec73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp" Nov 03 15:28:47.982983 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:47.982906 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca2f9ef-6560-4b12-a003-1980889cec73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db5e2ad8-kqdnp\" (UID: \"eca2f9ef-6560-4b12-a003-1980889cec73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp" Nov 03 15:28:47.983544 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:47.983522 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca2f9ef-6560-4b12-a003-1980889cec73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db5e2ad8-kqdnp\" (UID: \"eca2f9ef-6560-4b12-a003-1980889cec73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp" Nov 03 15:28:47.985146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:47.985127 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca2f9ef-6560-4b12-a003-1980889cec73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db5e2ad8-kqdnp\" (UID: \"eca2f9ef-6560-4b12-a003-1980889cec73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp" Nov 03 15:28:48.044113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:48.044077 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl826\" (UniqueName: \"kubernetes.io/projected/eca2f9ef-6560-4b12-a003-1980889cec73-kube-api-access-bl826\") pod \"maas-api-revoke-sub-db5e2ad8-kqdnp\" (UID: \"eca2f9ef-6560-4b12-a003-1980889cec73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp" Nov 03 15:28:48.050890 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:48.050864 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl"] Nov 03 15:28:48.054616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:48.054593 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617ae85d-x7trl"] Nov 03 15:28:48.065658 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:48.065630 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:28:48.139975 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:48.139902 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp" Nov 03 15:28:48.285305 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:48.285274 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="969b81a7-ad40-477e-9032-39e6532b16fd" path="/var/lib/kubelet/pods/969b81a7-ad40-477e-9032-39e6532b16fd/volumes" Nov 03 15:28:48.520137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:48.520102 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp"] Nov 03 15:28:48.520852 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:28:48.520829 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca2f9ef_6560_4b12_a003_1980889cec73.slice/crio-ea50e99c3449f6dfc9025529697338bf8cab1745592154cbd2ab69fdcf83f3d4 WatchSource:0}: Error finding container ea50e99c3449f6dfc9025529697338bf8cab1745592154cbd2ab69fdcf83f3d4: Status 404 returned error can't find the container with id ea50e99c3449f6dfc9025529697338bf8cab1745592154cbd2ab69fdcf83f3d4 Nov 03 15:28:49.273323 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:49.273283 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp" event={"ID":"eca2f9ef-6560-4b12-a003-1980889cec73","Type":"ContainerStarted","Data":"a6bb316625379253546aaf41154e99c3f9393ec909a7316f3b908b31ba9c615c"} Nov 03 15:28:49.273707 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:49.273332 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp" event={"ID":"eca2f9ef-6560-4b12-a003-1980889cec73","Type":"ContainerStarted","Data":"ea50e99c3449f6dfc9025529697338bf8cab1745592154cbd2ab69fdcf83f3d4"} Nov 03 15:28:49.416091 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:49.416043 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp" podStartSLOduration=2.4160307899999998 podStartE2EDuration="2.41603079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:28:49.411233497 +0000 UTC m=+1239.254355040" watchObservedRunningTime="2026-11-03 15:28:49.41603079 +0000 UTC m=+1239.259152331" Nov 03 15:28:50.280938 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:50.280905 4557 generic.go:356] "Generic (PLEG): container finished" podID="eca2f9ef-6560-4b12-a003-1980889cec73" containerID="a6bb316625379253546aaf41154e99c3f9393ec909a7316f3b908b31ba9c615c" exitCode=0 Nov 03 15:28:50.285777 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:50.285750 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp" event={"ID":"eca2f9ef-6560-4b12-a003-1980889cec73","Type":"ContainerDied","Data":"a6bb316625379253546aaf41154e99c3f9393ec909a7316f3b908b31ba9c615c"} Nov 03 15:28:51.458085 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:51.458060 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp" Nov 03 15:28:51.518126 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:51.518099 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eca2f9ef-6560-4b12-a003-1980889cec73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca2f9ef-6560-4b12-a003-1980889cec73-maas-api-openshift-ca\") pod \"eca2f9ef-6560-4b12-a003-1980889cec73\" (UID: \"eca2f9ef-6560-4b12-a003-1980889cec73\") " Nov 03 15:28:51.518259 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:51.518144 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eca2f9ef-6560-4b12-a003-1980889cec73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca2f9ef-6560-4b12-a003-1980889cec73-maas-api-xks-ca\") pod \"eca2f9ef-6560-4b12-a003-1980889cec73\" (UID: \"eca2f9ef-6560-4b12-a003-1980889cec73\") " Nov 03 15:28:51.518335 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:51.518313 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca2f9ef-6560-4b12-a003-1980889cec73-kube-api-access-bl826\" (UniqueName: \"kubernetes.io/projected/eca2f9ef-6560-4b12-a003-1980889cec73-kube-api-access-bl826\") pod \"eca2f9ef-6560-4b12-a003-1980889cec73\" (UID: \"eca2f9ef-6560-4b12-a003-1980889cec73\") " Nov 03 15:28:51.518496 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:51.518474 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca2f9ef-6560-4b12-a003-1980889cec73-maas-api-openshift-ca" pod "eca2f9ef-6560-4b12-a003-1980889cec73" (UID: "eca2f9ef-6560-4b12-a003-1980889cec73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:28:51.518772 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:51.518753 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca2f9ef-6560-4b12-a003-1980889cec73-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:28:51.520382 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:51.520356 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eca2f9ef-6560-4b12-a003-1980889cec73-maas-api-xks-ca" pod "eca2f9ef-6560-4b12-a003-1980889cec73" (UID: "eca2f9ef-6560-4b12-a003-1980889cec73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:28:51.520454 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:51.520384 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca2f9ef-6560-4b12-a003-1980889cec73-kube-api-access-bl826" pod "eca2f9ef-6560-4b12-a003-1980889cec73" (UID: "eca2f9ef-6560-4b12-a003-1980889cec73"). InnerVolumeSpecName "kube-api-access-bl826". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:28:51.620048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:51.620015 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl826\" (UniqueName: \"kubernetes.io/projected/eca2f9ef-6560-4b12-a003-1980889cec73-kube-api-access-bl826\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:28:51.620048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:51.620044 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca2f9ef-6560-4b12-a003-1980889cec73-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:28:52.293958 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:52.293930 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp" Nov 03 15:28:52.294113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:52.293935 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp" event={"ID":"eca2f9ef-6560-4b12-a003-1980889cec73","Type":"ContainerDied","Data":"ea50e99c3449f6dfc9025529697338bf8cab1745592154cbd2ab69fdcf83f3d4"} Nov 03 15:28:52.294113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:52.294098 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea50e99c3449f6dfc9025529697338bf8cab1745592154cbd2ab69fdcf83f3d4" Nov 03 15:28:53.212523 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:53.212488 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:28:54.065713 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:54.065678 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz"] Nov 03 15:28:54.111445 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:54.111405 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb88ad8c-wxfgz"] Nov 03 15:28:54.285343 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:54.285309 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5864b770-7acf-4498-a7fd-2319cabe6307" path="/var/lib/kubelet/pods/5864b770-7acf-4498-a7fd-2319cabe6307/volumes" Nov 03 15:28:57.921770 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:57.921740 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph"] Nov 03 15:28:57.922212 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:57.922199 4557 cpu_manager.go:414] "containerMap: removing container" podUID="eca2f9ef-6560-4b12-a003-1980889cec73" containerName="revoke-keys" Nov 03 15:28:57.922256 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:57.922214 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca2f9ef-6560-4b12-a003-1980889cec73" containerName="revoke-keys" Nov 03 15:28:57.922320 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:57.922311 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca2f9ef-6560-4b12-a003-1980889cec73" containerName="revoke-keys" Nov 03 15:28:57.925845 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:57.925829 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph" Nov 03 15:28:57.984559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:57.984531 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph"] Nov 03 15:28:57.993298 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:57.993275 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e32b0d-6883-42fd-96a4-4a966ded53f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-763c22db-vn5ph\" (UID: \"18e32b0d-6883-42fd-96a4-4a966ded53f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph" Nov 03 15:28:57.993400 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:57.993315 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e32b0d-6883-42fd-96a4-4a966ded53f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-763c22db-vn5ph\" (UID: \"18e32b0d-6883-42fd-96a4-4a966ded53f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph" Nov 03 15:28:57.993450 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:57.993428 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljtcz\" (UniqueName: \"kubernetes.io/projected/18e32b0d-6883-42fd-96a4-4a966ded53f4-kube-api-access-ljtcz\") pod \"maas-api-revoke-sub-763c22db-vn5ph\" (UID: \"18e32b0d-6883-42fd-96a4-4a966ded53f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph" Nov 03 15:28:58.029682 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:58.029657 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:28:58.029828 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:58.029751 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:28:58.094428 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:58.094395 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljtcz\" (UniqueName: \"kubernetes.io/projected/18e32b0d-6883-42fd-96a4-4a966ded53f4-kube-api-access-ljtcz\") pod \"maas-api-revoke-sub-763c22db-vn5ph\" (UID: \"18e32b0d-6883-42fd-96a4-4a966ded53f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph" Nov 03 15:28:58.094608 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:58.094525 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e32b0d-6883-42fd-96a4-4a966ded53f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-763c22db-vn5ph\" (UID: \"18e32b0d-6883-42fd-96a4-4a966ded53f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph" Nov 03 15:28:58.094608 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:58.094545 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e32b0d-6883-42fd-96a4-4a966ded53f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-763c22db-vn5ph\" (UID: \"18e32b0d-6883-42fd-96a4-4a966ded53f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph" Nov 03 15:28:58.095159 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:58.095138 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e32b0d-6883-42fd-96a4-4a966ded53f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-763c22db-vn5ph\" (UID: \"18e32b0d-6883-42fd-96a4-4a966ded53f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph" Nov 03 15:28:58.096819 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:58.096797 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e32b0d-6883-42fd-96a4-4a966ded53f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-763c22db-vn5ph\" (UID: \"18e32b0d-6883-42fd-96a4-4a966ded53f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph" Nov 03 15:28:58.156597 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:58.156559 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljtcz\" (UniqueName: \"kubernetes.io/projected/18e32b0d-6883-42fd-96a4-4a966ded53f4-kube-api-access-ljtcz\") pod \"maas-api-revoke-sub-763c22db-vn5ph\" (UID: \"18e32b0d-6883-42fd-96a4-4a966ded53f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph" Nov 03 15:28:58.207742 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:58.207562 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:28:58.240146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:58.240109 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph" Nov 03 15:28:58.635308 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:58.635273 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph"] Nov 03 15:28:58.635875 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:28:58.635850 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e32b0d_6883_42fd_96a4_4a966ded53f4.slice/crio-4d1a1e0a6121476172cb8dc91e826e43680229024e2e774a59f539476897e4c9 WatchSource:0}: Error finding container 4d1a1e0a6121476172cb8dc91e826e43680229024e2e774a59f539476897e4c9: Status 404 returned error can't find the container with id 4d1a1e0a6121476172cb8dc91e826e43680229024e2e774a59f539476897e4c9 Nov 03 15:28:59.341774 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:59.341738 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph" event={"ID":"18e32b0d-6883-42fd-96a4-4a966ded53f4","Type":"ContainerStarted","Data":"4d051d6c223a769287659c75f2e7be092cc8e9ff8927044001699a5ae7a33726"} Nov 03 15:28:59.342181 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:59.341800 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph" event={"ID":"18e32b0d-6883-42fd-96a4-4a966ded53f4","Type":"ContainerStarted","Data":"4d1a1e0a6121476172cb8dc91e826e43680229024e2e774a59f539476897e4c9"} Nov 03 15:28:59.461218 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:28:59.461176 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph" podStartSLOduration=2.461162599 podStartE2EDuration="2.461162599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:28:59.458229097 +0000 UTC m=+1249.301350641" watchObservedRunningTime="2026-11-03 15:28:59.461162599 +0000 UTC m=+1249.304284141" Nov 03 15:29:00.350407 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:00.350372 4557 generic.go:356] "Generic (PLEG): container finished" podID="18e32b0d-6883-42fd-96a4-4a966ded53f4" containerID="4d051d6c223a769287659c75f2e7be092cc8e9ff8927044001699a5ae7a33726" exitCode=0 Nov 03 15:29:00.350803 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:00.350427 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph" event={"ID":"18e32b0d-6883-42fd-96a4-4a966ded53f4","Type":"ContainerDied","Data":"4d051d6c223a769287659c75f2e7be092cc8e9ff8927044001699a5ae7a33726"} Nov 03 15:29:01.490036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.490012 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph" Nov 03 15:29:01.633010 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.632928 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e32b0d-6883-42fd-96a4-4a966ded53f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e32b0d-6883-42fd-96a4-4a966ded53f4-maas-api-openshift-ca\") pod \"18e32b0d-6883-42fd-96a4-4a966ded53f4\" (UID: \"18e32b0d-6883-42fd-96a4-4a966ded53f4\") " Nov 03 15:29:01.633181 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.633141 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18e32b0d-6883-42fd-96a4-4a966ded53f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e32b0d-6883-42fd-96a4-4a966ded53f4-maas-api-xks-ca\") pod \"18e32b0d-6883-42fd-96a4-4a966ded53f4\" (UID: \"18e32b0d-6883-42fd-96a4-4a966ded53f4\") " Nov 03 15:29:01.633250 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.633193 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e32b0d-6883-42fd-96a4-4a966ded53f4-kube-api-access-ljtcz\" (UniqueName: \"kubernetes.io/projected/18e32b0d-6883-42fd-96a4-4a966ded53f4-kube-api-access-ljtcz\") pod \"18e32b0d-6883-42fd-96a4-4a966ded53f4\" (UID: \"18e32b0d-6883-42fd-96a4-4a966ded53f4\") " Nov 03 15:29:01.633333 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.633312 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e32b0d-6883-42fd-96a4-4a966ded53f4-maas-api-openshift-ca" pod "18e32b0d-6883-42fd-96a4-4a966ded53f4" (UID: "18e32b0d-6883-42fd-96a4-4a966ded53f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:29:01.633631 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.633613 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e32b0d-6883-42fd-96a4-4a966ded53f4-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:01.635228 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.635203 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e32b0d-6883-42fd-96a4-4a966ded53f4-maas-api-xks-ca" pod "18e32b0d-6883-42fd-96a4-4a966ded53f4" (UID: "18e32b0d-6883-42fd-96a4-4a966ded53f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:29:01.635390 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.635367 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e32b0d-6883-42fd-96a4-4a966ded53f4-kube-api-access-ljtcz" pod "18e32b0d-6883-42fd-96a4-4a966ded53f4" (UID: "18e32b0d-6883-42fd-96a4-4a966ded53f4"). InnerVolumeSpecName "kube-api-access-ljtcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:29:01.674049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.674011 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8"] Nov 03 15:29:01.675127 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.675101 4557 cpu_manager.go:414] "containerMap: removing container" podUID="18e32b0d-6883-42fd-96a4-4a966ded53f4" containerName="revoke-keys" Nov 03 15:29:01.675127 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.675126 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e32b0d-6883-42fd-96a4-4a966ded53f4" containerName="revoke-keys" Nov 03 15:29:01.675265 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.675220 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e32b0d-6883-42fd-96a4-4a966ded53f4" containerName="revoke-keys" Nov 03 15:29:01.678571 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.678547 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.734881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.734851 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e32b0d-6883-42fd-96a4-4a966ded53f4-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:01.734881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.734877 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljtcz\" (UniqueName: \"kubernetes.io/projected/18e32b0d-6883-42fd-96a4-4a966ded53f4-kube-api-access-ljtcz\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:01.763448 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.763411 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8"] Nov 03 15:29:01.835513 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.835477 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.835694 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.835537 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.835694 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.835644 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53cb6efb-85fe-4c12-962c-436a99fedfa5-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.835773 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.835718 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.835773 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.835751 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.835867 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.835853 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbvl6\" (UniqueName: \"kubernetes.io/projected/53cb6efb-85fe-4c12-962c-436a99fedfa5-kube-api-access-qbvl6\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.835902 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.835884 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.835995 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.835982 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.836039 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.836027 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.937190 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.937093 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.937190 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.937179 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.937416 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.937216 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53cb6efb-85fe-4c12-962c-436a99fedfa5-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.937416 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.937243 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.937416 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.937272 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.937416 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.937330 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbvl6\" (UniqueName: \"kubernetes.io/projected/53cb6efb-85fe-4c12-962c-436a99fedfa5-kube-api-access-qbvl6\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.937416 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.937368 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.937694 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.937495 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.937694 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.937542 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.937799 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.937739 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.937855 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.937832 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.937951 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.937921 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.938020 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.937978 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.938173 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.938156 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53cb6efb-85fe-4c12-962c-436a99fedfa5-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.939492 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.939475 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:01.939856 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:01.939831 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:02.004071 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.004039 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:02.004406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.004386 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbvl6\" (UniqueName: \"kubernetes.io/projected/53cb6efb-85fe-4c12-962c-436a99fedfa5-kube-api-access-qbvl6\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:02.292084 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.292002 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:02.299740 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.299715 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:29:02.375432 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.375391 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph" event={"ID":"18e32b0d-6883-42fd-96a4-4a966ded53f4","Type":"ContainerDied","Data":"4d1a1e0a6121476172cb8dc91e826e43680229024e2e774a59f539476897e4c9"} Nov 03 15:29:02.375432 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.375410 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph" Nov 03 15:29:02.375432 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.375438 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d1a1e0a6121476172cb8dc91e826e43680229024e2e774a59f539476897e4c9" Nov 03 15:29:02.497149 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.497080 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:29:02.641379 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.641342 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m"] Nov 03 15:29:02.646410 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.646384 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m" Nov 03 15:29:02.704493 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.704458 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:29:02.705180 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.705135 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:29:02.708281 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.708256 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m"] Nov 03 15:29:02.746940 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:29:02.746894 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53cb6efb_85fe_4c12_962c_436a99fedfa5.slice/crio-b26def6448f397781d37d11338e4dcc1deec225bc433e6ba249f93f70792b13b WatchSource:0}: Error finding container b26def6448f397781d37d11338e4dcc1deec225bc433e6ba249f93f70792b13b: Status 404 returned error can't find the container with id b26def6448f397781d37d11338e4dcc1deec225bc433e6ba249f93f70792b13b Nov 03 15:29:02.747354 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.747326 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72f7302e-hk98m\" (UID: \"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m" Nov 03 15:29:02.747630 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.747558 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj5mm\" (UniqueName: \"kubernetes.io/projected/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-kube-api-access-cj5mm\") pod \"maas-api-revoke-sub-72f7302e-hk98m\" (UID: \"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m" Nov 03 15:29:02.747821 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.747797 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72f7302e-hk98m\" (UID: \"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m" Nov 03 15:29:02.749808 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.749778 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:29:02.749959 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.749859 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:29:02.749959 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.749904 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:29:02.758261 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.758237 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8"] Nov 03 15:29:02.848684 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.848649 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72f7302e-hk98m\" (UID: \"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m" Nov 03 15:29:02.848865 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.848772 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72f7302e-hk98m\" (UID: \"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m" Nov 03 15:29:02.848865 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.848793 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj5mm\" (UniqueName: \"kubernetes.io/projected/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-kube-api-access-cj5mm\") pod \"maas-api-revoke-sub-72f7302e-hk98m\" (UID: \"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m" Nov 03 15:29:02.849686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.849659 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72f7302e-hk98m\" (UID: \"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m" Nov 03 15:29:02.851669 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.851643 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72f7302e-hk98m\" (UID: \"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m" Nov 03 15:29:02.914508 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.914434 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj5mm\" (UniqueName: \"kubernetes.io/projected/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-kube-api-access-cj5mm\") pod \"maas-api-revoke-sub-72f7302e-hk98m\" (UID: \"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m" Nov 03 15:29:02.963366 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:02.963338 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m" Nov 03 15:29:03.292989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:03.292893 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:29:03.345998 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:29:03.345970 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0f07ad_9dc3_4251_b7b4_8a9d676f11b5.slice/crio-00154e3e3c12c803c73ad4d0a866236bcbd50e0a7e6a57ebf3485c66a5affef8 WatchSource:0}: Error finding container 00154e3e3c12c803c73ad4d0a866236bcbd50e0a7e6a57ebf3485c66a5affef8: Status 404 returned error can't find the container with id 00154e3e3c12c803c73ad4d0a866236bcbd50e0a7e6a57ebf3485c66a5affef8 Nov 03 15:29:03.346294 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:03.346272 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m"] Nov 03 15:29:03.409250 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:03.409193 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" event={"ID":"53cb6efb-85fe-4c12-962c-436a99fedfa5","Type":"ContainerStarted","Data":"711e472abcac085b515eeb310992e7697a4a08b1e5c7157acb66f3d500de8703"} Nov 03 15:29:03.409426 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:03.409270 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" event={"ID":"53cb6efb-85fe-4c12-962c-436a99fedfa5","Type":"ContainerStarted","Data":"b26def6448f397781d37d11338e4dcc1deec225bc433e6ba249f93f70792b13b"} Nov 03 15:29:03.411735 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:03.411707 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m" event={"ID":"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5","Type":"ContainerStarted","Data":"00154e3e3c12c803c73ad4d0a866236bcbd50e0a7e6a57ebf3485c66a5affef8"} Nov 03 15:29:03.533411 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:03.533348 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" podStartSLOduration=2.533332332 podStartE2EDuration="2.533332332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:29:03.531399346 +0000 UTC m=+1253.374520899" watchObservedRunningTime="2026-11-03 15:29:03.533332332 +0000 UTC m=+1253.376453873" Nov 03 15:29:03.972201 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:03.972112 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl"] Nov 03 15:29:03.976082 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:03.976062 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl" Nov 03 15:29:04.034523 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:04.034486 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl"] Nov 03 15:29:04.063084 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:04.063044 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bce079c1-zrbbl\" (UID: \"1e538a54-9596-4967-bfdb-a1acc7fe6aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl" Nov 03 15:29:04.063240 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:04.063198 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bce079c1-zrbbl\" (UID: \"1e538a54-9596-4967-bfdb-a1acc7fe6aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl" Nov 03 15:29:04.063386 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:04.063365 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6xfd\" (UniqueName: \"kubernetes.io/projected/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-kube-api-access-f6xfd\") pod \"maas-api-revoke-sub-bce079c1-zrbbl\" (UID: \"1e538a54-9596-4967-bfdb-a1acc7fe6aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl" Nov 03 15:29:04.164239 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:04.164206 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6xfd\" (UniqueName: \"kubernetes.io/projected/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-kube-api-access-f6xfd\") pod \"maas-api-revoke-sub-bce079c1-zrbbl\" (UID: \"1e538a54-9596-4967-bfdb-a1acc7fe6aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl" Nov 03 15:29:04.164460 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:04.164290 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bce079c1-zrbbl\" (UID: \"1e538a54-9596-4967-bfdb-a1acc7fe6aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl" Nov 03 15:29:04.164460 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:04.164357 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bce079c1-zrbbl\" (UID: \"1e538a54-9596-4967-bfdb-a1acc7fe6aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl" Nov 03 15:29:04.165022 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:04.164996 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bce079c1-zrbbl\" (UID: \"1e538a54-9596-4967-bfdb-a1acc7fe6aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl" Nov 03 15:29:04.166675 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:04.166648 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bce079c1-zrbbl\" (UID: \"1e538a54-9596-4967-bfdb-a1acc7fe6aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl" Nov 03 15:29:04.227736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:04.227667 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6xfd\" (UniqueName: \"kubernetes.io/projected/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-kube-api-access-f6xfd\") pod \"maas-api-revoke-sub-bce079c1-zrbbl\" (UID: \"1e538a54-9596-4967-bfdb-a1acc7fe6aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl" Nov 03 15:29:04.300606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:04.298313 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:04.302721 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:04.301723 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 15:29:04.302721 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:04.301779 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" podUID="53cb6efb-85fe-4c12-962c-436a99fedfa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 15:29:04.312382 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:04.312354 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl" Nov 03 15:29:04.436418 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:04.436371 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m" event={"ID":"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5","Type":"ContainerStarted","Data":"c551cc89aa396a5bb1b45e929e2c8ea5c331dff3b04aa492f14b7b651cafe575"} Nov 03 15:29:04.571454 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:04.571276 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m" podStartSLOduration=2.5712603659999997 podStartE2EDuration="2.571260366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:29:04.570983737 +0000 UTC m=+1254.414105279" watchObservedRunningTime="2026-11-03 15:29:04.571260366 +0000 UTC m=+1254.414381909" Nov 03 15:29:04.706962 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:29:04.706927 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e538a54_9596_4967_bfdb_a1acc7fe6aa5.slice/crio-df70db299f99a966fff87e0f06136620d97965563605227d901377e0cfcadd19 WatchSource:0}: Error finding container df70db299f99a966fff87e0f06136620d97965563605227d901377e0cfcadd19: Status 404 returned error can't find the container with id df70db299f99a966fff87e0f06136620d97965563605227d901377e0cfcadd19 Nov 03 15:29:04.707875 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:04.707850 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl"] Nov 03 15:29:05.301740 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:05.301702 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 15:29:05.302056 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:05.301770 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" podUID="53cb6efb-85fe-4c12-962c-436a99fedfa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 15:29:05.444364 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:05.444331 4557 generic.go:356] "Generic (PLEG): container finished" podID="9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5" containerID="c551cc89aa396a5bb1b45e929e2c8ea5c331dff3b04aa492f14b7b651cafe575" exitCode=0 Nov 03 15:29:05.444560 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:05.444423 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m" event={"ID":"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5","Type":"ContainerDied","Data":"c551cc89aa396a5bb1b45e929e2c8ea5c331dff3b04aa492f14b7b651cafe575"} Nov 03 15:29:05.446137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:05.446108 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl" event={"ID":"1e538a54-9596-4967-bfdb-a1acc7fe6aa5","Type":"ContainerStarted","Data":"9c9a43ff450ba3b75da729d10a05d5fef8bee2af32b30db984ac2168d03ae58f"} Nov 03 15:29:05.446250 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:05.446159 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl" event={"ID":"1e538a54-9596-4967-bfdb-a1acc7fe6aa5","Type":"ContainerStarted","Data":"df70db299f99a966fff87e0f06136620d97965563605227d901377e0cfcadd19"} Nov 03 15:29:06.292904 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:06.292868 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 15:29:06.293471 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:06.293437 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" podUID="53cb6efb-85fe-4c12-962c-436a99fedfa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 15:29:06.454253 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:06.454220 4557 generic.go:356] "Generic (PLEG): container finished" podID="1e538a54-9596-4967-bfdb-a1acc7fe6aa5" containerID="9c9a43ff450ba3b75da729d10a05d5fef8bee2af32b30db984ac2168d03ae58f" exitCode=0 Nov 03 15:29:06.454440 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:06.454270 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl" event={"ID":"1e538a54-9596-4967-bfdb-a1acc7fe6aa5","Type":"ContainerDied","Data":"9c9a43ff450ba3b75da729d10a05d5fef8bee2af32b30db984ac2168d03ae58f"} Nov 03 15:29:06.652713 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:06.652684 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m" Nov 03 15:29:06.823267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:06.823225 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-maas-api-openshift-ca\") pod \"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5\" (UID: \"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5\") " Nov 03 15:29:06.823460 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:06.823273 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-kube-api-access-cj5mm\" (UniqueName: \"kubernetes.io/projected/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-kube-api-access-cj5mm\") pod \"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5\" (UID: \"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5\") " Nov 03 15:29:06.823460 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:06.823367 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-maas-api-xks-ca\") pod \"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5\" (UID: \"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5\") " Nov 03 15:29:06.823684 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:06.823647 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-maas-api-openshift-ca" pod "9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5" (UID: "9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:29:06.823993 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:06.823970 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:06.825968 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:06.825943 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-maas-api-xks-ca" pod "9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5" (UID: "9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:29:06.826051 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:06.826003 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-kube-api-access-cj5mm" pod "9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5" (UID: "9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5"). InnerVolumeSpecName "kube-api-access-cj5mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:29:06.925150 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:06.925065 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj5mm\" (UniqueName: \"kubernetes.io/projected/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-kube-api-access-cj5mm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:06.925150 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:06.925094 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:07.293659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:07.292938 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 15:29:07.293659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:07.293001 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" podUID="53cb6efb-85fe-4c12-962c-436a99fedfa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 15:29:07.462910 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:07.462881 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m" Nov 03 15:29:07.463089 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:07.462878 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m" event={"ID":"9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5","Type":"ContainerDied","Data":"00154e3e3c12c803c73ad4d0a866236bcbd50e0a7e6a57ebf3485c66a5affef8"} Nov 03 15:29:07.463089 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:07.463010 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00154e3e3c12c803c73ad4d0a866236bcbd50e0a7e6a57ebf3485c66a5affef8" Nov 03 15:29:07.712653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:07.712629 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl" Nov 03 15:29:07.737267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:07.737239 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-maas-api-xks-ca\") pod \"1e538a54-9596-4967-bfdb-a1acc7fe6aa5\" (UID: \"1e538a54-9596-4967-bfdb-a1acc7fe6aa5\") " Nov 03 15:29:07.737430 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:07.737280 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-maas-api-openshift-ca\") pod \"1e538a54-9596-4967-bfdb-a1acc7fe6aa5\" (UID: \"1e538a54-9596-4967-bfdb-a1acc7fe6aa5\") " Nov 03 15:29:07.737430 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:07.737362 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-kube-api-access-f6xfd\" (UniqueName: \"kubernetes.io/projected/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-kube-api-access-f6xfd\") pod \"1e538a54-9596-4967-bfdb-a1acc7fe6aa5\" (UID: \"1e538a54-9596-4967-bfdb-a1acc7fe6aa5\") " Nov 03 15:29:07.737653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:07.737632 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-maas-api-openshift-ca" pod "1e538a54-9596-4967-bfdb-a1acc7fe6aa5" (UID: "1e538a54-9596-4967-bfdb-a1acc7fe6aa5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:29:07.737846 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:07.737830 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:07.739449 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:07.739428 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-kube-api-access-f6xfd" pod "1e538a54-9596-4967-bfdb-a1acc7fe6aa5" (UID: "1e538a54-9596-4967-bfdb-a1acc7fe6aa5"). InnerVolumeSpecName "kube-api-access-f6xfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:29:07.739534 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:07.739447 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-maas-api-xks-ca" pod "1e538a54-9596-4967-bfdb-a1acc7fe6aa5" (UID: "1e538a54-9596-4967-bfdb-a1acc7fe6aa5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:29:07.839208 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:07.839126 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:07.839208 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:07.839155 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6xfd\" (UniqueName: \"kubernetes.io/projected/1e538a54-9596-4967-bfdb-a1acc7fe6aa5-kube-api-access-f6xfd\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:08.300662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:08.293671 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 15:29:08.300662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:08.293729 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" podUID="53cb6efb-85fe-4c12-962c-436a99fedfa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 15:29:08.471189 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:08.471155 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl" Nov 03 15:29:08.471357 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:08.471187 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl" event={"ID":"1e538a54-9596-4967-bfdb-a1acc7fe6aa5","Type":"ContainerDied","Data":"df70db299f99a966fff87e0f06136620d97965563605227d901377e0cfcadd19"} Nov 03 15:29:08.471357 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:08.471258 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df70db299f99a966fff87e0f06136620d97965563605227d901377e0cfcadd19" Nov 03 15:29:08.655529 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:08.655454 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:29:08.852612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:08.852550 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:29:09.292889 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:09.292855 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 15:29:09.293096 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:09.292914 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" podUID="53cb6efb-85fe-4c12-962c-436a99fedfa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 15:29:10.057713 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:10.057670 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d"] Nov 03 15:29:10.107221 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:10.107174 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823513b6-5nl9d"] Nov 03 15:29:10.286675 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:10.286643 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8eeed72-3d78-49e0-9e5d-9a62420298b9" path="/var/lib/kubelet/pods/d8eeed72-3d78-49e0-9e5d-9a62420298b9/volumes" Nov 03 15:29:10.297240 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:10.297217 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:10.297511 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:10.297492 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:10.298141 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:10.298122 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:29:10.416342 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:10.416266 4557 scope.go:122] "RemoveContainer" containerID="ec5cfaa6ed47176f7ca19ae02c33e9c41b8424e00b48fcb4e74e4b9a1b89ac60" Nov 03 15:29:13.415711 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.415672 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd"] Nov 03 15:29:13.416338 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.416316 4557 cpu_manager.go:414] "containerMap: removing container" podUID="9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5" containerName="revoke-keys" Nov 03 15:29:13.416438 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.416339 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5" containerName="revoke-keys" Nov 03 15:29:13.416438 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.416361 4557 cpu_manager.go:414] "containerMap: removing container" podUID="1e538a54-9596-4967-bfdb-a1acc7fe6aa5" containerName="revoke-keys" Nov 03 15:29:13.416438 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.416369 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e538a54-9596-4967-bfdb-a1acc7fe6aa5" containerName="revoke-keys" Nov 03 15:29:13.416623 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.416539 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5" containerName="revoke-keys" Nov 03 15:29:13.416623 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.416553 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e538a54-9596-4967-bfdb-a1acc7fe6aa5" containerName="revoke-keys" Nov 03 15:29:13.421403 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.421383 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd" Nov 03 15:29:13.478172 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.478140 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd"] Nov 03 15:29:13.509720 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.509685 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75040bd4-blpsd\" (UID: \"fb75610c-d1f1-43aa-a0f0-27c3c438aed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd" Nov 03 15:29:13.509924 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.509851 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt7bm\" (UniqueName: \"kubernetes.io/projected/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-kube-api-access-qt7bm\") pod \"maas-api-revoke-sub-75040bd4-blpsd\" (UID: \"fb75610c-d1f1-43aa-a0f0-27c3c438aed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd" Nov 03 15:29:13.510081 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.510060 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75040bd4-blpsd\" (UID: \"fb75610c-d1f1-43aa-a0f0-27c3c438aed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd" Nov 03 15:29:13.523722 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.523696 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:29:13.523835 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.523765 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:29:13.611350 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.611316 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75040bd4-blpsd\" (UID: \"fb75610c-d1f1-43aa-a0f0-27c3c438aed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd" Nov 03 15:29:13.611518 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.611366 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75040bd4-blpsd\" (UID: \"fb75610c-d1f1-43aa-a0f0-27c3c438aed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd" Nov 03 15:29:13.611518 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.611436 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt7bm\" (UniqueName: \"kubernetes.io/projected/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-kube-api-access-qt7bm\") pod \"maas-api-revoke-sub-75040bd4-blpsd\" (UID: \"fb75610c-d1f1-43aa-a0f0-27c3c438aed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd" Nov 03 15:29:13.612246 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.612216 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75040bd4-blpsd\" (UID: \"fb75610c-d1f1-43aa-a0f0-27c3c438aed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd" Nov 03 15:29:13.613886 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.613861 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75040bd4-blpsd\" (UID: \"fb75610c-d1f1-43aa-a0f0-27c3c438aed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd" Nov 03 15:29:13.616614 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.616572 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:29:13.673290 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.673214 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt7bm\" (UniqueName: \"kubernetes.io/projected/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-kube-api-access-qt7bm\") pod \"maas-api-revoke-sub-75040bd4-blpsd\" (UID: \"fb75610c-d1f1-43aa-a0f0-27c3c438aed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd" Nov 03 15:29:13.734445 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:13.734415 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd" Nov 03 15:29:14.109524 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:14.109491 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd"] Nov 03 15:29:14.109808 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:29:14.109786 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb75610c_d1f1_43aa_a0f0_27c3c438aed2.slice/crio-7178fe5bf5cddecd76071be500d1efc67581612163ac6cc547859ff5beb00138 WatchSource:0}: Error finding container 7178fe5bf5cddecd76071be500d1efc67581612163ac6cc547859ff5beb00138: Status 404 returned error can't find the container with id 7178fe5bf5cddecd76071be500d1efc67581612163ac6cc547859ff5beb00138 Nov 03 15:29:14.519806 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:14.519771 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd" event={"ID":"fb75610c-d1f1-43aa-a0f0-27c3c438aed2","Type":"ContainerStarted","Data":"f293c81635e6105e8a23601860f6360af71c896a3f532a7d064f4a6912ff0695"} Nov 03 15:29:14.520193 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:14.519833 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd" event={"ID":"fb75610c-d1f1-43aa-a0f0-27c3c438aed2","Type":"ContainerStarted","Data":"7178fe5bf5cddecd76071be500d1efc67581612163ac6cc547859ff5beb00138"} Nov 03 15:29:14.636437 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:14.636346 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd" podStartSLOduration=1.6363282799999999 podStartE2EDuration="1.63632828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:29:14.635384754 +0000 UTC m=+1264.478506297" watchObservedRunningTime="2026-11-03 15:29:14.63632828 +0000 UTC m=+1264.479449825" Nov 03 15:29:15.137268 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:15.137237 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k"] Nov 03 15:29:15.144613 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:15.144567 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k" Nov 03 15:29:15.200867 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:15.200810 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k"] Nov 03 15:29:15.230486 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:15.230452 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29eab3f2-kgj7k\" (UID: \"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k" Nov 03 15:29:15.230689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:15.230653 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29eab3f2-kgj7k\" (UID: \"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k" Nov 03 15:29:15.230827 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:15.230757 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhrj8\" (UniqueName: \"kubernetes.io/projected/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-kube-api-access-lhrj8\") pod \"maas-api-revoke-sub-29eab3f2-kgj7k\" (UID: \"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k" Nov 03 15:29:15.331965 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:15.331931 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrj8\" (UniqueName: \"kubernetes.io/projected/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-kube-api-access-lhrj8\") pod \"maas-api-revoke-sub-29eab3f2-kgj7k\" (UID: \"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k" Nov 03 15:29:15.332131 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:15.332070 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29eab3f2-kgj7k\" (UID: \"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k" Nov 03 15:29:15.332198 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:15.332180 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29eab3f2-kgj7k\" (UID: \"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k" Nov 03 15:29:15.332660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:15.332639 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29eab3f2-kgj7k\" (UID: \"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k" Nov 03 15:29:15.334430 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:15.334408 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29eab3f2-kgj7k\" (UID: \"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k" Nov 03 15:29:15.393892 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:15.393868 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhrj8\" (UniqueName: \"kubernetes.io/projected/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-kube-api-access-lhrj8\") pod \"maas-api-revoke-sub-29eab3f2-kgj7k\" (UID: \"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k" Nov 03 15:29:15.458505 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:15.458447 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k" Nov 03 15:29:15.538245 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:15.538211 4557 generic.go:356] "Generic (PLEG): container finished" podID="fb75610c-d1f1-43aa-a0f0-27c3c438aed2" containerID="f293c81635e6105e8a23601860f6360af71c896a3f532a7d064f4a6912ff0695" exitCode=0 Nov 03 15:29:15.538687 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:15.538303 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd" event={"ID":"fb75610c-d1f1-43aa-a0f0-27c3c438aed2","Type":"ContainerDied","Data":"f293c81635e6105e8a23601860f6360af71c896a3f532a7d064f4a6912ff0695"} Nov 03 15:29:15.837314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:15.837287 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k"] Nov 03 15:29:15.837530 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:29:15.837511 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf4b5f3_ecd6_44a3_bf3d_b17bc716071a.slice/crio-7b449e4e198ba4ee27390f50b99db56571b04ef667fae592ae42e6cab9bb5f88 WatchSource:0}: Error finding container 7b449e4e198ba4ee27390f50b99db56571b04ef667fae592ae42e6cab9bb5f88: Status 404 returned error can't find the container with id 7b449e4e198ba4ee27390f50b99db56571b04ef667fae592ae42e6cab9bb5f88 Nov 03 15:29:16.531198 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:29:16.531116 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:29:16.546748 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:16.546716 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k" event={"ID":"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a","Type":"ContainerStarted","Data":"945290b9996c9ec6b16b50389149b43f24c57f76209b71f2a5eb602feacca3cc"} Nov 03 15:29:16.547114 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:16.546768 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k" event={"ID":"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a","Type":"ContainerStarted","Data":"7b449e4e198ba4ee27390f50b99db56571b04ef667fae592ae42e6cab9bb5f88"} Nov 03 15:29:16.681601 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:16.675411 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k" podStartSLOduration=1.675391246 podStartE2EDuration="1.675391246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:29:16.67096322 +0000 UTC m=+1266.514084775" watchObservedRunningTime="2026-11-03 15:29:16.675391246 +0000 UTC m=+1266.518512788" Nov 03 15:29:16.773557 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:16.773532 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd" Nov 03 15:29:16.855475 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:16.855442 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-maas-api-xks-ca\") pod \"fb75610c-d1f1-43aa-a0f0-27c3c438aed2\" (UID: \"fb75610c-d1f1-43aa-a0f0-27c3c438aed2\") " Nov 03 15:29:16.855692 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:16.855510 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-maas-api-openshift-ca\") pod \"fb75610c-d1f1-43aa-a0f0-27c3c438aed2\" (UID: \"fb75610c-d1f1-43aa-a0f0-27c3c438aed2\") " Nov 03 15:29:16.855776 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:16.855744 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-kube-api-access-qt7bm\" (UniqueName: \"kubernetes.io/projected/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-kube-api-access-qt7bm\") pod \"fb75610c-d1f1-43aa-a0f0-27c3c438aed2\" (UID: \"fb75610c-d1f1-43aa-a0f0-27c3c438aed2\") " Nov 03 15:29:16.856170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:16.856134 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-maas-api-openshift-ca" pod "fb75610c-d1f1-43aa-a0f0-27c3c438aed2" (UID: "fb75610c-d1f1-43aa-a0f0-27c3c438aed2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:29:16.872604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:16.858559 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-maas-api-xks-ca" pod "fb75610c-d1f1-43aa-a0f0-27c3c438aed2" (UID: "fb75610c-d1f1-43aa-a0f0-27c3c438aed2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:29:16.872604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:16.858598 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-kube-api-access-qt7bm" pod "fb75610c-d1f1-43aa-a0f0-27c3c438aed2" (UID: "fb75610c-d1f1-43aa-a0f0-27c3c438aed2"). InnerVolumeSpecName "kube-api-access-qt7bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:29:16.948555 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:16.948470 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:29:16.957426 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:16.957394 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt7bm\" (UniqueName: \"kubernetes.io/projected/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-kube-api-access-qt7bm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:16.957662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:16.957644 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:16.957765 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:16.957752 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb75610c-d1f1-43aa-a0f0-27c3c438aed2-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:17.105568 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.105495 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl"] Nov 03 15:29:17.106012 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.105997 4557 cpu_manager.go:414] "containerMap: removing container" podUID="fb75610c-d1f1-43aa-a0f0-27c3c438aed2" containerName="revoke-keys" Nov 03 15:29:17.106073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.106013 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb75610c-d1f1-43aa-a0f0-27c3c438aed2" containerName="revoke-keys" Nov 03 15:29:17.106127 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.106121 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb75610c-d1f1-43aa-a0f0-27c3c438aed2" containerName="revoke-keys" Nov 03 15:29:17.109749 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.109729 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.159977 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.159941 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.160390 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.160366 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.160648 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.160630 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.160858 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.160842 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.161036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.161005 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.161222 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.161192 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.161343 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.161327 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr5tn\" (UniqueName: \"kubernetes.io/projected/a4ea4afc-4c65-49fb-a234-b1d695a8a868-kube-api-access-rr5tn\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.161513 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.161486 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.161798 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.161780 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.203605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.200876 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl"] Nov 03 15:29:17.263693 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.263653 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.263904 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.263740 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.263904 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.263769 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr5tn\" (UniqueName: \"kubernetes.io/projected/a4ea4afc-4c65-49fb-a234-b1d695a8a868-kube-api-access-rr5tn\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.263904 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.263844 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.264083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.263913 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.264083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.263958 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.264083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.263981 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.264083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.264071 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.264216 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.264162 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.264789 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.264738 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.264905 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.264800 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.265048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.265026 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.265048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.265038 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.265247 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.265226 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.266711 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.266689 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.268074 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.268028 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.326832 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.326796 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.327155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.327111 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr5tn\" (UniqueName: \"kubernetes.io/projected/a4ea4afc-4c65-49fb-a234-b1d695a8a868-kube-api-access-rr5tn\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.424214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.423898 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:17.649036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.648464 4557 generic.go:356] "Generic (PLEG): container finished" podID="ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a" containerID="945290b9996c9ec6b16b50389149b43f24c57f76209b71f2a5eb602feacca3cc" exitCode=0 Nov 03 15:29:17.649036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.648637 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k" event={"ID":"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a","Type":"ContainerDied","Data":"945290b9996c9ec6b16b50389149b43f24c57f76209b71f2a5eb602feacca3cc"} Nov 03 15:29:17.714656 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.711408 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd" event={"ID":"fb75610c-d1f1-43aa-a0f0-27c3c438aed2","Type":"ContainerDied","Data":"7178fe5bf5cddecd76071be500d1efc67581612163ac6cc547859ff5beb00138"} Nov 03 15:29:17.714656 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.711471 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7178fe5bf5cddecd76071be500d1efc67581612163ac6cc547859ff5beb00138" Nov 03 15:29:17.714656 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:17.711548 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd" Nov 03 15:29:18.565868 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:18.565809 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:18.637974 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:18.637927 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:18.656462 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:18.656414 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:18.699208 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:18.699163 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:18.709785 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:18.709745 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:18.894664 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:18.860516 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl"] Nov 03 15:29:18.896949 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:18.867816 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:18.897214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:18.895329 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:29:18.998615 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:29:18.970140 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ea4afc_4c65_49fb_a234_b1d695a8a868.slice/crio-b3c660ac4a20e3ab2ecf0f4abdd7be70ffbb87f6bb6e06353376560cf6ff88d1 WatchSource:0}: Error finding container b3c660ac4a20e3ab2ecf0f4abdd7be70ffbb87f6bb6e06353376560cf6ff88d1: Status 404 returned error can't find the container with id b3c660ac4a20e3ab2ecf0f4abdd7be70ffbb87f6bb6e06353376560cf6ff88d1 Nov 03 15:29:18.998615 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:18.975176 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:29:18.998615 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:18.975291 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:29:18.998615 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:18.975356 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:29:19.188726 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:19.188663 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:19.284603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:19.284541 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 15:29:19.356607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:19.356230 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:19.541025 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:19.534639 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 15:29:19.667651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:19.658594 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:19.673341 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:19.673301 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:19.677482 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:19.677435 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:29:19.690331 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:19.682269 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 15:29:19.690331 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:19.688847 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:29:19.775421 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:19.775318 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:19.775831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:19.775806 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:29:19.810256 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:19.810226 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" event={"ID":"a4ea4afc-4c65-49fb-a234-b1d695a8a868","Type":"ContainerStarted","Data":"6279aff283b48494bc0cc0a6142299ab44fd9faa0fefdcc04157a01f0d1390ba"} Nov 03 15:29:19.810860 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:19.810836 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" event={"ID":"a4ea4afc-4c65-49fb-a234-b1d695a8a868","Type":"ContainerStarted","Data":"b3c660ac4a20e3ab2ecf0f4abdd7be70ffbb87f6bb6e06353376560cf6ff88d1"} Nov 03 15:29:19.918524 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:19.918479 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 03 15:29:19.919222 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:19.919198 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:29:19.925606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:19.924963 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k" Nov 03 15:29:20.141602 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.141442 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:29:20.141602 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.141516 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:29:20.156119 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.155699 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:29:20.172928 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.172896 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj"] Nov 03 15:29:20.188606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.184768 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:29:20.214253 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.213055 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-maas-api-xks-ca\") pod \"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a\" (UID: \"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a\") " Nov 03 15:29:20.214253 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.213104 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-maas-api-openshift-ca\") pod \"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a\" (UID: \"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a\") " Nov 03 15:29:20.214253 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.213178 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-kube-api-access-lhrj8\" (UniqueName: \"kubernetes.io/projected/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-kube-api-access-lhrj8\") pod \"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a\" (UID: \"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a\") " Nov 03 15:29:20.214820 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.214786 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-maas-api-openshift-ca" pod "ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a" (UID: "ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:29:20.220996 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.220968 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-kube-api-access-lhrj8" pod "ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a" (UID: "ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a"). InnerVolumeSpecName "kube-api-access-lhrj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:29:20.258104 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.258049 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-maas-api-xks-ca" pod "ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a" (UID: "ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:29:20.265887 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.265825 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76f64878-psglj"] Nov 03 15:29:20.328366 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.328314 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhrj8\" (UniqueName: \"kubernetes.io/projected/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-kube-api-access-lhrj8\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:20.328552 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.328380 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:20.328552 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.328426 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:20.334879 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.334845 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f383e1be-904d-40d8-8f33-df320987b7cb" path="/var/lib/kubelet/pods/f383e1be-904d-40d8-8f33-df320987b7cb/volumes" Nov 03 15:29:20.427143 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.426537 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:20.430019 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.428680 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 15:29:20.430019 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.428729 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" podUID="a4ea4afc-4c65-49fb-a234-b1d695a8a868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 15:29:20.710355 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.710239 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" podStartSLOduration=3.710218648 podStartE2EDuration="3.710218648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:29:20.700232887 +0000 UTC m=+1270.543354430" watchObservedRunningTime="2026-11-03 15:29:20.710218648 +0000 UTC m=+1270.553340191" Nov 03 15:29:20.851781 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.851749 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k" Nov 03 15:29:20.859466 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.854222 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k" event={"ID":"ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a","Type":"ContainerDied","Data":"7b449e4e198ba4ee27390f50b99db56571b04ef667fae592ae42e6cab9bb5f88"} Nov 03 15:29:20.859466 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:20.854266 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b449e4e198ba4ee27390f50b99db56571b04ef667fae592ae42e6cab9bb5f88" Nov 03 15:29:21.150598 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.150149 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9"] Nov 03 15:29:21.150917 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.150892 4557 cpu_manager.go:414] "containerMap: removing container" podUID="ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a" containerName="revoke-keys" Nov 03 15:29:21.150917 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.150913 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a" containerName="revoke-keys" Nov 03 15:29:21.151127 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.151056 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a" containerName="revoke-keys" Nov 03 15:29:21.156096 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.155563 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9" Nov 03 15:29:21.215606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.215225 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:29:21.219605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.215934 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:29:21.236419 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.235435 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9"] Nov 03 15:29:21.262172 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.262141 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnrxz\" (UniqueName: \"kubernetes.io/projected/73ace764-2928-4065-be34-16d73be85112-kube-api-access-vnrxz\") pod \"maas-api-revoke-sub-663878af-lfnw9\" (UID: \"73ace764-2928-4065-be34-16d73be85112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9" Nov 03 15:29:21.262339 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.262301 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ace764-2928-4065-be34-16d73be85112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-663878af-lfnw9\" (UID: \"73ace764-2928-4065-be34-16d73be85112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9" Nov 03 15:29:21.262381 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.262344 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ace764-2928-4065-be34-16d73be85112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-663878af-lfnw9\" (UID: \"73ace764-2928-4065-be34-16d73be85112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9" Nov 03 15:29:21.363347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.363307 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnrxz\" (UniqueName: \"kubernetes.io/projected/73ace764-2928-4065-be34-16d73be85112-kube-api-access-vnrxz\") pod \"maas-api-revoke-sub-663878af-lfnw9\" (UID: \"73ace764-2928-4065-be34-16d73be85112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9" Nov 03 15:29:21.363524 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.363508 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ace764-2928-4065-be34-16d73be85112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-663878af-lfnw9\" (UID: \"73ace764-2928-4065-be34-16d73be85112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9" Nov 03 15:29:21.363617 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.363596 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ace764-2928-4065-be34-16d73be85112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-663878af-lfnw9\" (UID: \"73ace764-2928-4065-be34-16d73be85112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9" Nov 03 15:29:21.364210 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.364188 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ace764-2928-4065-be34-16d73be85112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-663878af-lfnw9\" (UID: \"73ace764-2928-4065-be34-16d73be85112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9" Nov 03 15:29:21.365930 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.365908 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ace764-2928-4065-be34-16d73be85112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-663878af-lfnw9\" (UID: \"73ace764-2928-4065-be34-16d73be85112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9" Nov 03 15:29:21.425227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.425151 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 15:29:21.425227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.425205 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" podUID="a4ea4afc-4c65-49fb-a234-b1d695a8a868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 15:29:21.425555 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.425528 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnrxz\" (UniqueName: \"kubernetes.io/projected/73ace764-2928-4065-be34-16d73be85112-kube-api-access-vnrxz\") pod \"maas-api-revoke-sub-663878af-lfnw9\" (UID: \"73ace764-2928-4065-be34-16d73be85112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9" Nov 03 15:29:21.472575 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.472550 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9" Nov 03 15:29:21.911767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:21.911729 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9"] Nov 03 15:29:22.425897 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:22.425862 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 15:29:22.426059 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:22.425930 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" podUID="a4ea4afc-4c65-49fb-a234-b1d695a8a868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 15:29:22.866319 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:22.866284 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9" event={"ID":"73ace764-2928-4065-be34-16d73be85112","Type":"ContainerStarted","Data":"66ae51747982d275392d6d7a682bb35e72a83742edc1e86b1ef661b7621532d9"} Nov 03 15:29:22.866475 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:22.866336 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9" event={"ID":"73ace764-2928-4065-be34-16d73be85112","Type":"ContainerStarted","Data":"e790ce10c26608ac51946102cf586f4ef436b1e63ee217cdf4f184f2016bb911"} Nov 03 15:29:23.425510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:23.425471 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 15:29:23.425932 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:23.425525 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" podUID="a4ea4afc-4c65-49fb-a234-b1d695a8a868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 15:29:23.775768 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:23.775687 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:29:23.875195 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:23.875161 4557 generic.go:356] "Generic (PLEG): container finished" podID="73ace764-2928-4065-be34-16d73be85112" containerID="66ae51747982d275392d6d7a682bb35e72a83742edc1e86b1ef661b7621532d9" exitCode=0 Nov 03 15:29:23.875389 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:23.875245 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9" event={"ID":"73ace764-2928-4065-be34-16d73be85112","Type":"ContainerDied","Data":"66ae51747982d275392d6d7a682bb35e72a83742edc1e86b1ef661b7621532d9"} Nov 03 15:29:24.425527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:24.425492 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 15:29:24.425936 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:24.425552 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" podUID="a4ea4afc-4c65-49fb-a234-b1d695a8a868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 15:29:25.072224 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:25.072200 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9" Nov 03 15:29:25.131571 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:25.131539 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ace764-2928-4065-be34-16d73be85112-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ace764-2928-4065-be34-16d73be85112-maas-api-openshift-ca\") pod \"73ace764-2928-4065-be34-16d73be85112\" (UID: \"73ace764-2928-4065-be34-16d73be85112\") " Nov 03 15:29:25.131766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:25.131747 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ace764-2928-4065-be34-16d73be85112-kube-api-access-vnrxz\" (UniqueName: \"kubernetes.io/projected/73ace764-2928-4065-be34-16d73be85112-kube-api-access-vnrxz\") pod \"73ace764-2928-4065-be34-16d73be85112\" (UID: \"73ace764-2928-4065-be34-16d73be85112\") " Nov 03 15:29:25.131844 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:25.131820 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73ace764-2928-4065-be34-16d73be85112-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ace764-2928-4065-be34-16d73be85112-maas-api-xks-ca\") pod \"73ace764-2928-4065-be34-16d73be85112\" (UID: \"73ace764-2928-4065-be34-16d73be85112\") " Nov 03 15:29:25.132233 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:25.131986 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ace764-2928-4065-be34-16d73be85112-maas-api-openshift-ca" pod "73ace764-2928-4065-be34-16d73be85112" (UID: "73ace764-2928-4065-be34-16d73be85112"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:29:25.132340 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:25.132296 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ace764-2928-4065-be34-16d73be85112-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:25.134004 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:25.133963 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ace764-2928-4065-be34-16d73be85112-kube-api-access-vnrxz" pod "73ace764-2928-4065-be34-16d73be85112" (UID: "73ace764-2928-4065-be34-16d73be85112"). InnerVolumeSpecName "kube-api-access-vnrxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:29:25.134100 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:25.133999 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ace764-2928-4065-be34-16d73be85112-maas-api-xks-ca" pod "73ace764-2928-4065-be34-16d73be85112" (UID: "73ace764-2928-4065-be34-16d73be85112"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:29:25.233799 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:25.233719 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnrxz\" (UniqueName: \"kubernetes.io/projected/73ace764-2928-4065-be34-16d73be85112-kube-api-access-vnrxz\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:25.233799 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:25.233746 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ace764-2928-4065-be34-16d73be85112-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:25.425822 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:25.425786 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 15:29:25.426308 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:25.425845 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" podUID="a4ea4afc-4c65-49fb-a234-b1d695a8a868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 15:29:25.913497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:25.913469 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9" Nov 03 15:29:25.913684 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:25.913508 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9" event={"ID":"73ace764-2928-4065-be34-16d73be85112","Type":"ContainerDied","Data":"e790ce10c26608ac51946102cf586f4ef436b1e63ee217cdf4f184f2016bb911"} Nov 03 15:29:25.913684 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:25.913601 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e790ce10c26608ac51946102cf586f4ef436b1e63ee217cdf4f184f2016bb911" Nov 03 15:29:26.425233 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:26.425194 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 15:29:26.425416 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:26.425262 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" podUID="a4ea4afc-4c65-49fb-a234-b1d695a8a868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 15:29:27.049762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:27.049730 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46"] Nov 03 15:29:27.102613 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:27.102562 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e80bd18-8dw46"] Nov 03 15:29:27.493497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:27.493414 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:27.493918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:27.493886 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:27.494676 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:27.494649 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:29:28.285545 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:28.285514 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f11ee03-17b1-433c-810b-3672a025a470" path="/var/lib/kubelet/pods/3f11ee03-17b1-433c-810b-3672a025a470/volumes" Nov 03 15:29:28.642910 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:28.642870 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:29:30.296880 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.296849 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c"] Nov 03 15:29:30.297307 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.297294 4557 cpu_manager.go:414] "containerMap: removing container" podUID="73ace764-2928-4065-be34-16d73be85112" containerName="revoke-keys" Nov 03 15:29:30.297361 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.297307 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ace764-2928-4065-be34-16d73be85112" containerName="revoke-keys" Nov 03 15:29:30.297401 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.297386 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ace764-2928-4065-be34-16d73be85112" containerName="revoke-keys" Nov 03 15:29:30.300386 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.300371 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c" Nov 03 15:29:30.359759 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.359733 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c"] Nov 03 15:29:30.393910 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.393874 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e860c079-fab1-4d54-b509-fe47cb98d018-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26134315-4zj5c\" (UID: \"e860c079-fab1-4d54-b509-fe47cb98d018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c" Nov 03 15:29:30.394065 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.393958 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e860c079-fab1-4d54-b509-fe47cb98d018-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26134315-4zj5c\" (UID: \"e860c079-fab1-4d54-b509-fe47cb98d018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c" Nov 03 15:29:30.394159 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.394131 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlfgg\" (UniqueName: \"kubernetes.io/projected/e860c079-fab1-4d54-b509-fe47cb98d018-kube-api-access-rlfgg\") pod \"maas-api-revoke-sub-26134315-4zj5c\" (UID: \"e860c079-fab1-4d54-b509-fe47cb98d018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c" Nov 03 15:29:30.404737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.404711 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:29:30.404737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.404729 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:29:30.494944 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.494922 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e860c079-fab1-4d54-b509-fe47cb98d018-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26134315-4zj5c\" (UID: \"e860c079-fab1-4d54-b509-fe47cb98d018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c" Nov 03 15:29:30.495116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.494990 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlfgg\" (UniqueName: \"kubernetes.io/projected/e860c079-fab1-4d54-b509-fe47cb98d018-kube-api-access-rlfgg\") pod \"maas-api-revoke-sub-26134315-4zj5c\" (UID: \"e860c079-fab1-4d54-b509-fe47cb98d018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c" Nov 03 15:29:30.495116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.495092 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e860c079-fab1-4d54-b509-fe47cb98d018-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26134315-4zj5c\" (UID: \"e860c079-fab1-4d54-b509-fe47cb98d018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c" Nov 03 15:29:30.495528 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.495508 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e860c079-fab1-4d54-b509-fe47cb98d018-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26134315-4zj5c\" (UID: \"e860c079-fab1-4d54-b509-fe47cb98d018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c" Nov 03 15:29:30.497381 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.497358 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e860c079-fab1-4d54-b509-fe47cb98d018-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26134315-4zj5c\" (UID: \"e860c079-fab1-4d54-b509-fe47cb98d018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c" Nov 03 15:29:30.556461 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.556406 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlfgg\" (UniqueName: \"kubernetes.io/projected/e860c079-fab1-4d54-b509-fe47cb98d018-kube-api-access-rlfgg\") pod \"maas-api-revoke-sub-26134315-4zj5c\" (UID: \"e860c079-fab1-4d54-b509-fe47cb98d018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c" Nov 03 15:29:30.613262 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.613237 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c" Nov 03 15:29:30.951071 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.950999 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll"] Nov 03 15:29:30.955801 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.955780 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll" Nov 03 15:29:30.995099 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:30.995071 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c"] Nov 03 15:29:30.995210 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:29:30.995109 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode860c079_fab1_4d54_b509_fe47cb98d018.slice/crio-40677a37edb1493a67da9e73c1deb16a95e2749a1491b2bc10947df37043ffba WatchSource:0}: Error finding container 40677a37edb1493a67da9e73c1deb16a95e2749a1491b2bc10947df37043ffba: Status 404 returned error can't find the container with id 40677a37edb1493a67da9e73c1deb16a95e2749a1491b2bc10947df37043ffba Nov 03 15:29:31.000885 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:31.000864 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6192a7dc-9bkll\" (UID: \"6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll" Nov 03 15:29:31.000997 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:31.000983 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghzr4\" (UniqueName: \"kubernetes.io/projected/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-kube-api-access-ghzr4\") pod \"maas-api-revoke-sub-6192a7dc-9bkll\" (UID: \"6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll" Nov 03 15:29:31.001064 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:31.001052 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6192a7dc-9bkll\" (UID: \"6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll" Nov 03 15:29:31.015119 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:31.015092 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll"] Nov 03 15:29:31.102084 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:31.102051 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6192a7dc-9bkll\" (UID: \"6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll" Nov 03 15:29:31.102211 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:31.102142 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6192a7dc-9bkll\" (UID: \"6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll" Nov 03 15:29:31.102247 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:31.102206 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghzr4\" (UniqueName: \"kubernetes.io/projected/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-kube-api-access-ghzr4\") pod \"maas-api-revoke-sub-6192a7dc-9bkll\" (UID: \"6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll" Nov 03 15:29:31.102735 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:31.102709 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6192a7dc-9bkll\" (UID: \"6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll" Nov 03 15:29:31.104402 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:31.104380 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6192a7dc-9bkll\" (UID: \"6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll" Nov 03 15:29:31.164359 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:31.164329 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghzr4\" (UniqueName: \"kubernetes.io/projected/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-kube-api-access-ghzr4\") pod \"maas-api-revoke-sub-6192a7dc-9bkll\" (UID: \"6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll" Nov 03 15:29:31.270879 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:31.270798 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll" Nov 03 15:29:31.649593 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:29:31.649545 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6370e1ab_ec68_4de8_b8a0_7b9cb3a75aae.slice/crio-e3d5c8f7bd72fe51b45bd535dac94f70a1ed66d05313616b26e54f0aed54b3aa WatchSource:0}: Error finding container e3d5c8f7bd72fe51b45bd535dac94f70a1ed66d05313616b26e54f0aed54b3aa: Status 404 returned error can't find the container with id e3d5c8f7bd72fe51b45bd535dac94f70a1ed66d05313616b26e54f0aed54b3aa Nov 03 15:29:31.649898 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:31.649792 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll"] Nov 03 15:29:31.956431 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:31.956400 4557 generic.go:356] "Generic (PLEG): container finished" podID="e860c079-fab1-4d54-b509-fe47cb98d018" containerID="edc5beccb4be7b7404817ca859fc846301f037bf8d259584d97cf04d16b0d7f4" exitCode=0 Nov 03 15:29:31.956568 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:31.956493 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c" event={"ID":"e860c079-fab1-4d54-b509-fe47cb98d018","Type":"ContainerDied","Data":"edc5beccb4be7b7404817ca859fc846301f037bf8d259584d97cf04d16b0d7f4"} Nov 03 15:29:31.956568 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:31.956550 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c" event={"ID":"e860c079-fab1-4d54-b509-fe47cb98d018","Type":"ContainerStarted","Data":"40677a37edb1493a67da9e73c1deb16a95e2749a1491b2bc10947df37043ffba"} Nov 03 15:29:31.957845 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:31.957826 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll" event={"ID":"6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae","Type":"ContainerStarted","Data":"e3d5c8f7bd72fe51b45bd535dac94f70a1ed66d05313616b26e54f0aed54b3aa"} Nov 03 15:29:32.967210 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:32.967173 4557 generic.go:356] "Generic (PLEG): container finished" podID="6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae" containerID="eea172ff772d874ee99a3bf8cd3399b3061a638ed9e06991ce3cea48edfa1c77" exitCode=0 Nov 03 15:29:32.967630 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:32.967263 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll" event={"ID":"6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae","Type":"ContainerDied","Data":"eea172ff772d874ee99a3bf8cd3399b3061a638ed9e06991ce3cea48edfa1c77"} Nov 03 15:29:33.075822 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:33.075791 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4"] Nov 03 15:29:33.081414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:33.081381 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-596be1cd-kdgk4"] Nov 03 15:29:33.124297 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:33.124276 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c" Nov 03 15:29:33.230790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:33.230705 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e860c079-fab1-4d54-b509-fe47cb98d018-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e860c079-fab1-4d54-b509-fe47cb98d018-maas-api-xks-ca\") pod \"e860c079-fab1-4d54-b509-fe47cb98d018\" (UID: \"e860c079-fab1-4d54-b509-fe47cb98d018\") " Nov 03 15:29:33.230948 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:33.230836 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e860c079-fab1-4d54-b509-fe47cb98d018-kube-api-access-rlfgg\" (UniqueName: \"kubernetes.io/projected/e860c079-fab1-4d54-b509-fe47cb98d018-kube-api-access-rlfgg\") pod \"e860c079-fab1-4d54-b509-fe47cb98d018\" (UID: \"e860c079-fab1-4d54-b509-fe47cb98d018\") " Nov 03 15:29:33.230948 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:33.230877 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e860c079-fab1-4d54-b509-fe47cb98d018-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e860c079-fab1-4d54-b509-fe47cb98d018-maas-api-openshift-ca\") pod \"e860c079-fab1-4d54-b509-fe47cb98d018\" (UID: \"e860c079-fab1-4d54-b509-fe47cb98d018\") " Nov 03 15:29:33.231316 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:33.231286 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e860c079-fab1-4d54-b509-fe47cb98d018-maas-api-openshift-ca" pod "e860c079-fab1-4d54-b509-fe47cb98d018" (UID: "e860c079-fab1-4d54-b509-fe47cb98d018"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:29:33.232890 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:33.232866 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e860c079-fab1-4d54-b509-fe47cb98d018-maas-api-xks-ca" pod "e860c079-fab1-4d54-b509-fe47cb98d018" (UID: "e860c079-fab1-4d54-b509-fe47cb98d018"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:29:33.233041 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:33.233023 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e860c079-fab1-4d54-b509-fe47cb98d018-kube-api-access-rlfgg" pod "e860c079-fab1-4d54-b509-fe47cb98d018" (UID: "e860c079-fab1-4d54-b509-fe47cb98d018"). InnerVolumeSpecName "kube-api-access-rlfgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:29:33.332673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:33.332646 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlfgg\" (UniqueName: \"kubernetes.io/projected/e860c079-fab1-4d54-b509-fe47cb98d018-kube-api-access-rlfgg\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:33.332824 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:33.332782 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e860c079-fab1-4d54-b509-fe47cb98d018-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:33.332824 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:33.332811 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e860c079-fab1-4d54-b509-fe47cb98d018-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:33.505070 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:33.504964 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:29:33.976191 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:33.976167 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c" Nov 03 15:29:33.976546 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:33.976198 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c" event={"ID":"e860c079-fab1-4d54-b509-fe47cb98d018","Type":"ContainerDied","Data":"40677a37edb1493a67da9e73c1deb16a95e2749a1491b2bc10947df37043ffba"} Nov 03 15:29:33.976546 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:33.976246 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40677a37edb1493a67da9e73c1deb16a95e2749a1491b2bc10947df37043ffba" Nov 03 15:29:34.109264 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:34.109243 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll" Nov 03 15:29:34.243796 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:34.243707 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-maas-api-xks-ca\") pod \"6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae\" (UID: \"6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae\") " Nov 03 15:29:34.243796 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:34.243774 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-kube-api-access-ghzr4\" (UniqueName: \"kubernetes.io/projected/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-kube-api-access-ghzr4\") pod \"6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae\" (UID: \"6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae\") " Nov 03 15:29:34.244038 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:34.243918 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-maas-api-openshift-ca\") pod \"6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae\" (UID: \"6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae\") " Nov 03 15:29:34.244243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:34.244208 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-maas-api-openshift-ca" pod "6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae" (UID: "6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:29:34.245957 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:34.245934 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-maas-api-xks-ca" pod "6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae" (UID: "6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:29:34.245957 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:34.245942 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-kube-api-access-ghzr4" pod "6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae" (UID: "6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae"). InnerVolumeSpecName "kube-api-access-ghzr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:29:34.285815 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:34.285786 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f3ed764-4fb3-46b1-88b6-1dc1dcff3154" path="/var/lib/kubelet/pods/2f3ed764-4fb3-46b1-88b6-1dc1dcff3154/volumes" Nov 03 15:29:34.345213 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:34.345183 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:34.345213 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:34.345210 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:34.345399 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:34.345221 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghzr4\" (UniqueName: \"kubernetes.io/projected/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae-kube-api-access-ghzr4\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:34.983789 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:34.983762 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll" Nov 03 15:29:34.984373 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:34.983802 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll" event={"ID":"6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae","Type":"ContainerDied","Data":"e3d5c8f7bd72fe51b45bd535dac94f70a1ed66d05313616b26e54f0aed54b3aa"} Nov 03 15:29:34.984373 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:34.983854 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3d5c8f7bd72fe51b45bd535dac94f70a1ed66d05313616b26e54f0aed54b3aa" Nov 03 15:29:38.366384 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:38.366344 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:29:41.050613 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:41.050564 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn"] Nov 03 15:29:41.099509 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:41.099479 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732560b8-9hxpn"] Nov 03 15:29:42.285703 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:42.285673 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b" path="/var/lib/kubelet/pods/eeee83d4-4b43-4cc1-8c9c-85b5b32fe57b/volumes" Nov 03 15:29:43.309555 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:43.309515 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:29:45.051123 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:45.051086 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8"] Nov 03 15:29:45.100954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:45.100920 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c780ce0-9ptc8"] Nov 03 15:29:46.016209 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.016175 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t"] Nov 03 15:29:46.016674 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.016660 4557 cpu_manager.go:414] "containerMap: removing container" podUID="e860c079-fab1-4d54-b509-fe47cb98d018" containerName="revoke-keys" Nov 03 15:29:46.016725 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.016675 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e860c079-fab1-4d54-b509-fe47cb98d018" containerName="revoke-keys" Nov 03 15:29:46.016725 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.016692 4557 cpu_manager.go:414] "containerMap: removing container" podUID="6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae" containerName="revoke-keys" Nov 03 15:29:46.016725 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.016697 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae" containerName="revoke-keys" Nov 03 15:29:46.016823 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.016801 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="e860c079-fab1-4d54-b509-fe47cb98d018" containerName="revoke-keys" Nov 03 15:29:46.016823 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.016811 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae" containerName="revoke-keys" Nov 03 15:29:46.021433 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.021416 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.065752 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.065730 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf"] Nov 03 15:29:46.069500 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.069485 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.126539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.126507 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t"] Nov 03 15:29:46.176973 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.176942 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf"] Nov 03 15:29:46.181430 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.181399 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.181799 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.181768 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.181927 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.181814 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.181927 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.181860 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f4nm\" (UniqueName: \"kubernetes.io/projected/11e80ae6-ae0d-4857-b490-ed829f3b1412-kube-api-access-8f4nm\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.181927 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.181908 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.182047 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.181943 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.182047 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.181971 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.182047 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.182003 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.182047 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.182036 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.182185 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.182067 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee31efd-a014-4cfa-8578-bed65d98bfea-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.182185 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.182164 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.182258 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.182193 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.182258 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.182236 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt22r\" (UniqueName: \"kubernetes.io/projected/7ee31efd-a014-4cfa-8578-bed65d98bfea-kube-api-access-nt22r\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.182350 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.182285 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.182350 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.182309 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e80ae6-ae0d-4857-b490-ed829f3b1412-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.182439 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.182355 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.182439 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.182401 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.182439 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.182431 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.284043 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.283940 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.284043 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.283975 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.284043 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.284002 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt22r\" (UniqueName: \"kubernetes.io/projected/7ee31efd-a014-4cfa-8578-bed65d98bfea-kube-api-access-nt22r\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.284325 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.284169 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.284325 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.284214 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e80ae6-ae0d-4857-b490-ed829f3b1412-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.284325 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.284263 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.284325 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.284312 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.284523 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.284343 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.284523 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.284346 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.284523 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.284398 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.284523 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.284468 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.284523 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.284500 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.284778 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.284550 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f4nm\" (UniqueName: \"kubernetes.io/projected/11e80ae6-ae0d-4857-b490-ed829f3b1412-kube-api-access-8f4nm\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.284778 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.284617 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.284778 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.284657 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.284778 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.284683 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.284778 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.284684 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.284778 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.284741 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.284778 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.284780 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.285138 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.284817 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee31efd-a014-4cfa-8578-bed65d98bfea-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.285255 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.285226 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.285321 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.285277 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.285381 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.285308 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e80ae6-ae0d-4857-b490-ed829f3b1412-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.285543 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.285519 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.285769 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.285603 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee31efd-a014-4cfa-8578-bed65d98bfea-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.285881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.285601 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.285975 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.285805 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.286037 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.285830 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.286817 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.286795 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.286928 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.286906 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.286985 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.286960 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.287889 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.287867 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.287954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.287936 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b2a25f-7307-41c6-9fee-902e24f9ac3b" path="/var/lib/kubelet/pods/a9b2a25f-7307-41c6-9fee-902e24f9ac3b/volumes" Nov 03 15:29:46.345942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.345915 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt22r\" (UniqueName: \"kubernetes.io/projected/7ee31efd-a014-4cfa-8578-bed65d98bfea-kube-api-access-nt22r\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.352131 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.352100 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.356438 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.356416 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.371032 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.371010 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f4nm\" (UniqueName: \"kubernetes.io/projected/11e80ae6-ae0d-4857-b490-ed829f3b1412-kube-api-access-8f4nm\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.381868 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.381846 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:46.530435 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:29:46.530399 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:29:46.634264 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.634239 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:46.819288 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:29:46.819258 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e80ae6_ae0d_4857_b490_ed829f3b1412.slice/crio-ece96c5ade6bee976ddef78577147c236bfaaced011f733a7315bb456379aa45 WatchSource:0}: Error finding container ece96c5ade6bee976ddef78577147c236bfaaced011f733a7315bb456379aa45: Status 404 returned error can't find the container with id ece96c5ade6bee976ddef78577147c236bfaaced011f733a7315bb456379aa45 Nov 03 15:29:46.824870 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.821090 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf"] Nov 03 15:29:46.827818 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.827779 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:29:46.827895 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.827874 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:29:46.827951 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:46.827921 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:29:47.031653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.031616 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t"] Nov 03 15:29:47.037215 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:29:47.037186 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee31efd_a014_4cfa_8578_bed65d98bfea.slice/crio-868de35044f665c93ec50cf3bce85b122d3cb750344add898173dcdbc077d248 WatchSource:0}: Error finding container 868de35044f665c93ec50cf3bce85b122d3cb750344add898173dcdbc077d248: Status 404 returned error can't find the container with id 868de35044f665c93ec50cf3bce85b122d3cb750344add898173dcdbc077d248 Nov 03 15:29:47.039714 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.039686 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:29:47.039859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.039743 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:29:47.039859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.039770 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:29:47.055916 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.055891 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7"] Nov 03 15:29:47.061176 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.061150 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b45612a-sklh7"] Nov 03 15:29:47.065820 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.065792 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" event={"ID":"11e80ae6-ae0d-4857-b490-ed829f3b1412","Type":"ContainerStarted","Data":"4afc8d4cc8107a00d67d79d966dc4d75a6424485c8dd77e33cd3d23f404d8afb"} Nov 03 15:29:47.066150 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.065851 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" event={"ID":"11e80ae6-ae0d-4857-b490-ed829f3b1412","Type":"ContainerStarted","Data":"ece96c5ade6bee976ddef78577147c236bfaaced011f733a7315bb456379aa45"} Nov 03 15:29:47.067320 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.067300 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" event={"ID":"7ee31efd-a014-4cfa-8578-bed65d98bfea","Type":"ContainerStarted","Data":"868de35044f665c93ec50cf3bce85b122d3cb750344add898173dcdbc077d248"} Nov 03 15:29:47.200647 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.200436 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" podStartSLOduration=1.2004182079999999 podStartE2EDuration="1.200418208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:29:47.189102861 +0000 UTC m=+1297.032224403" watchObservedRunningTime="2026-11-03 15:29:47.200418208 +0000 UTC m=+1297.043539754" Nov 03 15:29:47.382863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.382826 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:47.384112 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.384087 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 15:29:47.384236 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.384135 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" podUID="11e80ae6-ae0d-4857-b490-ed829f3b1412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 15:29:47.556148 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.556116 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd"] Nov 03 15:29:47.560125 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.560108 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd" Nov 03 15:29:47.614072 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.614036 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:29:47.614248 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.614173 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:29:47.618787 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.618759 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd"] Nov 03 15:29:47.704766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.704731 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8e942a-184c-42d3-aa1b-660ca6884b66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d952a5ad-pqvpd\" (UID: \"0d8e942a-184c-42d3-aa1b-660ca6884b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd" Nov 03 15:29:47.704946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.704839 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxbkm\" (UniqueName: \"kubernetes.io/projected/0d8e942a-184c-42d3-aa1b-660ca6884b66-kube-api-access-cxbkm\") pod \"maas-api-revoke-sub-d952a5ad-pqvpd\" (UID: \"0d8e942a-184c-42d3-aa1b-660ca6884b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd" Nov 03 15:29:47.705007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.704941 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8e942a-184c-42d3-aa1b-660ca6884b66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d952a5ad-pqvpd\" (UID: \"0d8e942a-184c-42d3-aa1b-660ca6884b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd" Nov 03 15:29:47.807651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.806347 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8e942a-184c-42d3-aa1b-660ca6884b66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d952a5ad-pqvpd\" (UID: \"0d8e942a-184c-42d3-aa1b-660ca6884b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd" Nov 03 15:29:47.807651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.806568 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8e942a-184c-42d3-aa1b-660ca6884b66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d952a5ad-pqvpd\" (UID: \"0d8e942a-184c-42d3-aa1b-660ca6884b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd" Nov 03 15:29:47.807651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.806661 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxbkm\" (UniqueName: \"kubernetes.io/projected/0d8e942a-184c-42d3-aa1b-660ca6884b66-kube-api-access-cxbkm\") pod \"maas-api-revoke-sub-d952a5ad-pqvpd\" (UID: \"0d8e942a-184c-42d3-aa1b-660ca6884b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd" Nov 03 15:29:47.808206 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.808178 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8e942a-184c-42d3-aa1b-660ca6884b66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d952a5ad-pqvpd\" (UID: \"0d8e942a-184c-42d3-aa1b-660ca6884b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd" Nov 03 15:29:47.810527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.810493 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8e942a-184c-42d3-aa1b-660ca6884b66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d952a5ad-pqvpd\" (UID: \"0d8e942a-184c-42d3-aa1b-660ca6884b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd" Nov 03 15:29:47.900696 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:47.900661 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxbkm\" (UniqueName: \"kubernetes.io/projected/0d8e942a-184c-42d3-aa1b-660ca6884b66-kube-api-access-cxbkm\") pod \"maas-api-revoke-sub-d952a5ad-pqvpd\" (UID: \"0d8e942a-184c-42d3-aa1b-660ca6884b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd" Nov 03 15:29:48.102665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:48.095617 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" event={"ID":"7ee31efd-a014-4cfa-8578-bed65d98bfea","Type":"ContainerStarted","Data":"3e72e181b8c94e8d20c11df3afeb90ea2db355bf31b370ac23bb9daae0fa5687"} Nov 03 15:29:48.174712 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:48.174680 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd" Nov 03 15:29:48.229185 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:48.228162 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" podStartSLOduration=3.228138858 podStartE2EDuration="3.228138858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:29:48.203942888 +0000 UTC m=+1298.047064431" watchObservedRunningTime="2026-11-03 15:29:48.228138858 +0000 UTC m=+1298.071260401" Nov 03 15:29:48.335246 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:48.332095 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec209e9-6c77-4d08-801a-7195b980e41e" path="/var/lib/kubelet/pods/6ec209e9-6c77-4d08-801a-7195b980e41e/volumes" Nov 03 15:29:48.401668 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:48.390947 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 15:29:48.401668 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:48.391009 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" podUID="11e80ae6-ae0d-4857-b490-ed829f3b1412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 15:29:48.417601 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:48.407569 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:29:48.636607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:48.635060 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:48.636607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:48.636302 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 15:29:48.636607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:48.636345 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" podUID="7ee31efd-a014-4cfa-8578-bed65d98bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 15:29:48.738884 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:48.738856 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd"] Nov 03 15:29:48.741660 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:29:48.741611 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d8e942a_184c_42d3_aa1b_660ca6884b66.slice/crio-c1f7f76c4ef066a87d4694295ef8944c24a7eedf2ae1a308e5305cce3754a36a WatchSource:0}: Error finding container c1f7f76c4ef066a87d4694295ef8944c24a7eedf2ae1a308e5305cce3754a36a: Status 404 returned error can't find the container with id c1f7f76c4ef066a87d4694295ef8944c24a7eedf2ae1a308e5305cce3754a36a Nov 03 15:29:49.123613 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:49.122468 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd" event={"ID":"0d8e942a-184c-42d3-aa1b-660ca6884b66","Type":"ContainerStarted","Data":"c1f7f76c4ef066a87d4694295ef8944c24a7eedf2ae1a308e5305cce3754a36a"} Nov 03 15:29:49.372599 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:49.372511 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2"] Nov 03 15:29:49.398626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:49.398304 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 15:29:49.398626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:49.398369 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" podUID="11e80ae6-ae0d-4857-b490-ed829f3b1412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 15:29:49.433207 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:49.433176 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2" Nov 03 15:29:49.441575 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:49.441551 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2"] Nov 03 15:29:49.461664 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:49.461635 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24b30eb-e757-4180-883d-718ce48bf780-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97de7d34-7zbx2\" (UID: \"d24b30eb-e757-4180-883d-718ce48bf780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2" Nov 03 15:29:49.461788 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:49.461677 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24b30eb-e757-4180-883d-718ce48bf780-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97de7d34-7zbx2\" (UID: \"d24b30eb-e757-4180-883d-718ce48bf780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2" Nov 03 15:29:49.461901 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:49.461887 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5xkx\" (UniqueName: \"kubernetes.io/projected/d24b30eb-e757-4180-883d-718ce48bf780-kube-api-access-d5xkx\") pod \"maas-api-revoke-sub-97de7d34-7zbx2\" (UID: \"d24b30eb-e757-4180-883d-718ce48bf780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2" Nov 03 15:29:49.563311 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:49.563274 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5xkx\" (UniqueName: \"kubernetes.io/projected/d24b30eb-e757-4180-883d-718ce48bf780-kube-api-access-d5xkx\") pod \"maas-api-revoke-sub-97de7d34-7zbx2\" (UID: \"d24b30eb-e757-4180-883d-718ce48bf780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2" Nov 03 15:29:49.563476 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:49.563359 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24b30eb-e757-4180-883d-718ce48bf780-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97de7d34-7zbx2\" (UID: \"d24b30eb-e757-4180-883d-718ce48bf780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2" Nov 03 15:29:49.563476 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:49.563388 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24b30eb-e757-4180-883d-718ce48bf780-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97de7d34-7zbx2\" (UID: \"d24b30eb-e757-4180-883d-718ce48bf780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2" Nov 03 15:29:49.564060 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:49.564039 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24b30eb-e757-4180-883d-718ce48bf780-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97de7d34-7zbx2\" (UID: \"d24b30eb-e757-4180-883d-718ce48bf780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2" Nov 03 15:29:49.565739 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:49.565715 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24b30eb-e757-4180-883d-718ce48bf780-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97de7d34-7zbx2\" (UID: \"d24b30eb-e757-4180-883d-718ce48bf780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2" Nov 03 15:29:49.629323 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:49.629250 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5xkx\" (UniqueName: \"kubernetes.io/projected/d24b30eb-e757-4180-883d-718ce48bf780-kube-api-access-d5xkx\") pod \"maas-api-revoke-sub-97de7d34-7zbx2\" (UID: \"d24b30eb-e757-4180-883d-718ce48bf780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2" Nov 03 15:29:49.635109 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:49.635065 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 15:29:49.635216 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:49.635138 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" podUID="7ee31efd-a014-4cfa-8578-bed65d98bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 15:29:49.753770 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:49.753738 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2" Nov 03 15:29:50.050135 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:50.050095 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn"] Nov 03 15:29:50.056043 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:50.056017 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0114e22f-tmbsn"] Nov 03 15:29:50.141653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:50.141610 4557 generic.go:356] "Generic (PLEG): container finished" podID="0d8e942a-184c-42d3-aa1b-660ca6884b66" containerID="396d5213d238de9a2ee78c3c96c0848449dd74c2941aaa68800cb48645bf1267" exitCode=0 Nov 03 15:29:50.142045 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:50.141714 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd" event={"ID":"0d8e942a-184c-42d3-aa1b-660ca6884b66","Type":"ContainerDied","Data":"396d5213d238de9a2ee78c3c96c0848449dd74c2941aaa68800cb48645bf1267"} Nov 03 15:29:50.216534 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:50.216497 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2"] Nov 03 15:29:50.335479 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:50.335445 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac5cb6ab-283f-4ff1-a3ea-f8d958451962" path="/var/lib/kubelet/pods/ac5cb6ab-283f-4ff1-a3ea-f8d958451962/volumes" Nov 03 15:29:50.399606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:50.386680 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 15:29:50.399606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:50.386734 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" podUID="11e80ae6-ae0d-4857-b490-ed829f3b1412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 15:29:50.635978 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:50.635940 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 15:29:50.636254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:50.636226 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" podUID="7ee31efd-a014-4cfa-8578-bed65d98bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 15:29:51.229662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:51.219764 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2" event={"ID":"d24b30eb-e757-4180-883d-718ce48bf780","Type":"ContainerStarted","Data":"a97fbed3450487774d9e9115b3f40c2690362bf137a10095e370f62588aa7fa5"} Nov 03 15:29:51.229662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:51.219827 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2" event={"ID":"d24b30eb-e757-4180-883d-718ce48bf780","Type":"ContainerStarted","Data":"857dc81ec1cbbfd78396c2e1caaeebe563ee3efd7773e137ee37d11212a0fa6e"} Nov 03 15:29:51.346605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:51.346434 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2" podStartSLOduration=2.346413088 podStartE2EDuration="2.346413088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:29:51.34459687 +0000 UTC m=+1301.187718407" watchObservedRunningTime="2026-11-03 15:29:51.346413088 +0000 UTC m=+1301.189534631" Nov 03 15:29:51.386227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:51.386192 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 15:29:51.386515 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:51.386484 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" podUID="11e80ae6-ae0d-4857-b490-ed829f3b1412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 15:29:51.644605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:51.644040 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 15:29:51.644605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:51.644111 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" podUID="7ee31efd-a014-4cfa-8578-bed65d98bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 15:29:52.216841 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.216789 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 15:29:52.461658 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.217108 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 15:29:52.461658 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.217486 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 15:29:52.461658 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.219438 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Nov 03 15:29:52.461658 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.219468 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 03 15:29:52.461658 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.219488 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 15:29:52.461658 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.267700 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:52.551845 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.551799 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 15:29:52.557689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.557644 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" podUID="11e80ae6-ae0d-4857-b490-ed829f3b1412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 15:29:52.582606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.552447 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:52.584890 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.566686 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:52.584890 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.567571 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 03 15:29:52.608990 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.607536 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Nov 03 15:29:52.608990 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.608917 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded" Nov 03 15:29:52.649631 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.621439 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:52.653885 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.653846 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 15:29:52.654125 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.654097 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" podUID="7ee31efd-a014-4cfa-8578-bed65d98bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 15:29:52.685186 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.685149 4557 generic.go:356] "Generic (PLEG): container finished" podID="d24b30eb-e757-4180-883d-718ce48bf780" containerID="a97fbed3450487774d9e9115b3f40c2690362bf137a10095e370f62588aa7fa5" exitCode=0 Nov 03 15:29:52.687843 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.686718 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2" event={"ID":"d24b30eb-e757-4180-883d-718ce48bf780","Type":"ContainerDied","Data":"a97fbed3450487774d9e9115b3f40c2690362bf137a10095e370f62588aa7fa5"} Nov 03 15:29:52.945606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:52.931984 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd" Nov 03 15:29:53.007414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.006925 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d8e942a-184c-42d3-aa1b-660ca6884b66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8e942a-184c-42d3-aa1b-660ca6884b66-maas-api-openshift-ca\") pod \"0d8e942a-184c-42d3-aa1b-660ca6884b66\" (UID: \"0d8e942a-184c-42d3-aa1b-660ca6884b66\") " Nov 03 15:29:53.007761 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.007739 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d8e942a-184c-42d3-aa1b-660ca6884b66-kube-api-access-cxbkm\" (UniqueName: \"kubernetes.io/projected/0d8e942a-184c-42d3-aa1b-660ca6884b66-kube-api-access-cxbkm\") pod \"0d8e942a-184c-42d3-aa1b-660ca6884b66\" (UID: \"0d8e942a-184c-42d3-aa1b-660ca6884b66\") " Nov 03 15:29:53.008693 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.008334 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d8e942a-184c-42d3-aa1b-660ca6884b66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8e942a-184c-42d3-aa1b-660ca6884b66-maas-api-xks-ca\") pod \"0d8e942a-184c-42d3-aa1b-660ca6884b66\" (UID: \"0d8e942a-184c-42d3-aa1b-660ca6884b66\") " Nov 03 15:29:53.008987 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.007378 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8e942a-184c-42d3-aa1b-660ca6884b66-maas-api-openshift-ca" pod "0d8e942a-184c-42d3-aa1b-660ca6884b66" (UID: "0d8e942a-184c-42d3-aa1b-660ca6884b66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:29:53.009908 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.009888 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8e942a-184c-42d3-aa1b-660ca6884b66-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:53.062372 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.062152 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d8e942a-184c-42d3-aa1b-660ca6884b66-kube-api-access-cxbkm" pod "0d8e942a-184c-42d3-aa1b-660ca6884b66" (UID: "0d8e942a-184c-42d3-aa1b-660ca6884b66"). InnerVolumeSpecName "kube-api-access-cxbkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:29:53.072551 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.069817 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d8e942a-184c-42d3-aa1b-660ca6884b66-maas-api-xks-ca" pod "0d8e942a-184c-42d3-aa1b-660ca6884b66" (UID: "0d8e942a-184c-42d3-aa1b-660ca6884b66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:29:53.111149 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.110989 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxbkm\" (UniqueName: \"kubernetes.io/projected/0d8e942a-184c-42d3-aa1b-660ca6884b66-kube-api-access-cxbkm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:53.111149 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.111030 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8e942a-184c-42d3-aa1b-660ca6884b66-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:53.399942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.399850 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 15:29:53.400593 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.400514 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" podUID="11e80ae6-ae0d-4857-b490-ed829f3b1412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 15:29:53.447880 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.447839 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:53.448937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.448906 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Nov 03 15:29:53.449248 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.449215 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:53.449481 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.449459 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:53.449736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.449713 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:53.450553 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.450523 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 15:29:53.451028 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.451002 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded" Nov 03 15:29:53.451258 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.451242 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:29:53.451424 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.451409 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:29:53.451558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.451545 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:29:53.451705 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.451692 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:29:53.451859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.451846 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:29:53.452005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.451991 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:29:53.454754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.451257 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 15:29:53.492230 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.491457 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:29:53.682554 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.682440 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 03 15:29:53.683165 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.683133 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:53.683724 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.683705 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:29:53.701947 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.701919 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:29:53.703526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.684073 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:53.716905 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.716860 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:53.717375 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.717345 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 15:29:53.854132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.684145 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 15:29:53.854404 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.854368 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" podUID="7ee31efd-a014-4cfa-8578-bed65d98bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 15:29:53.968655 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:53.960323 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:29:54.475671 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.424711 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:29:54.475671 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.398014 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 15:29:54.475671 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.444685 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" podUID="11e80ae6-ae0d-4857-b490-ed829f3b1412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 15:29:54.475671 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.408655 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:29:54.475671 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.444764 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" podUID="5dc6676c-58c3-4ad9-9701-e76f70622bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:54.475671 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.446488 4557 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:29:54.475671 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.446610 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" podUID="b37f9ec1-b784-41fd-9a79-8df424aa77e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:54.475671 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.458703 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:54.475671 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.459113 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 15:29:54.475671 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.459457 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Nov 03 15:29:54.475671 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.459786 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 03 15:29:54.475671 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.460032 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 15:29:54.475671 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.460257 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 15:29:54.527630 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.511648 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 15:29:54.527630 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.527498 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:29:54.527630 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.527554 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:29:54.584675 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.584622 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd" Nov 03 15:29:54.600225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.588207 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd" event={"ID":"0d8e942a-184c-42d3-aa1b-660ca6884b66","Type":"ContainerDied","Data":"c1f7f76c4ef066a87d4694295ef8944c24a7eedf2ae1a308e5305cce3754a36a"} Nov 03 15:29:54.600225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.588262 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1f7f76c4ef066a87d4694295ef8944c24a7eedf2ae1a308e5305cce3754a36a" Nov 03 15:29:54.700542 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.695749 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:54.713161 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.703810 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 03 15:29:54.740178 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.730188 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 15:29:54.740178 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.730240 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" podUID="7ee31efd-a014-4cfa-8578-bed65d98bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 15:29:54.740478 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.740452 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:29:54.903738 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.903690 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:54.932368 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.931768 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:29:54.955022 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:54.954983 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:55.035355 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.035268 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:29:55.061814 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.061784 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:29:55.148605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.147669 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:55.367465 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.367356 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:29:55.416641 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.415765 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 15:29:55.416641 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.415839 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" podUID="11e80ae6-ae0d-4857-b490-ed829f3b1412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 15:29:55.479809 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.478371 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:55.502455 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.499650 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 15:29:55.502455 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.499956 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Nov 03 15:29:55.502455 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.500171 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 03 15:29:55.502455 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.500497 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:55.632676 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.590316 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:55.667607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.641723 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 15:29:55.667607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.641788 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" podUID="7ee31efd-a014-4cfa-8578-bed65d98bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 15:29:55.667607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.656714 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:29:55.667607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.656792 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" podUID="72be94c1-dd49-4f71-8a37-6056e0939ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:55.858604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.857738 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:55.878426 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.878388 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:55.879373 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.879340 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 15:29:55.960913 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:55.960706 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:56.134934 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.134818 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:56.467605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.395189 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 15:29:56.504811 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.442667 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:29:56.505170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.505149 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" podUID="53cb6efb-85fe-4c12-962c-436a99fedfa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:56.505382 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.505071 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" podUID="11e80ae6-ae0d-4857-b490-ed829f3b1412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 15:29:56.505621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.505531 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:56.505857 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.505832 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:29:56.506554 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.506523 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 03 15:29:56.507043 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.507005 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 15:29:56.507448 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.505333 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 15:29:56.834854 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.834799 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:29:56.847925 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.847894 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:29:56.856481 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.856453 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:29:56.860762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.857315 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 15:29:56.860762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.857377 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" podUID="7ee31efd-a014-4cfa-8578-bed65d98bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 15:29:56.860762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.857534 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:29:56.876964 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.876938 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:29:56.877267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.877233 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:29:56.878124 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.877846 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:29:56.905299 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.904856 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:29:56.905299 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.905204 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:29:56.906825 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.906800 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2" event={"ID":"d24b30eb-e757-4180-883d-718ce48bf780","Type":"ContainerDied","Data":"857dc81ec1cbbfd78396c2e1caaeebe563ee3efd7773e137ee37d11212a0fa6e"} Nov 03 15:29:56.906941 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.906846 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="857dc81ec1cbbfd78396c2e1caaeebe563ee3efd7773e137ee37d11212a0fa6e" Nov 03 15:29:56.909772 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:56.909750 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2" Nov 03 15:29:57.057445 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.057411 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24b30eb-e757-4180-883d-718ce48bf780-kube-api-access-d5xkx\" (UniqueName: \"kubernetes.io/projected/d24b30eb-e757-4180-883d-718ce48bf780-kube-api-access-d5xkx\") pod \"d24b30eb-e757-4180-883d-718ce48bf780\" (UID: \"d24b30eb-e757-4180-883d-718ce48bf780\") " Nov 03 15:29:57.057646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.057478 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d24b30eb-e757-4180-883d-718ce48bf780-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24b30eb-e757-4180-883d-718ce48bf780-maas-api-openshift-ca\") pod \"d24b30eb-e757-4180-883d-718ce48bf780\" (UID: \"d24b30eb-e757-4180-883d-718ce48bf780\") " Nov 03 15:29:57.057714 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.057690 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d24b30eb-e757-4180-883d-718ce48bf780-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24b30eb-e757-4180-883d-718ce48bf780-maas-api-xks-ca\") pod \"d24b30eb-e757-4180-883d-718ce48bf780\" (UID: \"d24b30eb-e757-4180-883d-718ce48bf780\") " Nov 03 15:29:57.057951 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.057925 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24b30eb-e757-4180-883d-718ce48bf780-maas-api-openshift-ca" pod "d24b30eb-e757-4180-883d-718ce48bf780" (UID: "d24b30eb-e757-4180-883d-718ce48bf780"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:29:57.059070 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.058780 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24b30eb-e757-4180-883d-718ce48bf780-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:57.060360 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.060339 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24b30eb-e757-4180-883d-718ce48bf780-kube-api-access-d5xkx" pod "d24b30eb-e757-4180-883d-718ce48bf780" (UID: "d24b30eb-e757-4180-883d-718ce48bf780"). InnerVolumeSpecName "kube-api-access-d5xkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:29:57.060758 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.060729 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d24b30eb-e757-4180-883d-718ce48bf780-maas-api-xks-ca" pod "d24b30eb-e757-4180-883d-718ce48bf780" (UID: "d24b30eb-e757-4180-883d-718ce48bf780"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:29:57.159972 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.159889 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5xkx\" (UniqueName: \"kubernetes.io/projected/d24b30eb-e757-4180-883d-718ce48bf780-kube-api-access-d5xkx\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:57.159972 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.159915 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24b30eb-e757-4180-883d-718ce48bf780-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:29:57.383039 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.383005 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 15:29:57.383221 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.383060 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" podUID="11e80ae6-ae0d-4857-b490-ed829f3b1412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 15:29:57.565324 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.565287 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq"] Nov 03 15:29:57.566219 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.566190 4557 cpu_manager.go:414] "containerMap: removing container" podUID="d24b30eb-e757-4180-883d-718ce48bf780" containerName="revoke-keys" Nov 03 15:29:57.566219 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.566216 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24b30eb-e757-4180-883d-718ce48bf780" containerName="revoke-keys" Nov 03 15:29:57.566219 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.566276 4557 cpu_manager.go:414] "containerMap: removing container" podUID="0d8e942a-184c-42d3-aa1b-660ca6884b66" containerName="revoke-keys" Nov 03 15:29:57.566525 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.566285 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d8e942a-184c-42d3-aa1b-660ca6884b66" containerName="revoke-keys" Nov 03 15:29:57.570420 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.570372 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d8e942a-184c-42d3-aa1b-660ca6884b66" containerName="revoke-keys" Nov 03 15:29:57.572387 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.570730 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24b30eb-e757-4180-883d-718ce48bf780" containerName="revoke-keys" Nov 03 15:29:57.576643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.576616 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq" Nov 03 15:29:57.629096 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.629054 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq"] Nov 03 15:29:57.635269 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.635242 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 15:29:57.635413 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.635298 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" podUID="7ee31efd-a014-4cfa-8578-bed65d98bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 15:29:57.667365 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.667329 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a0b8bc-24a3-4450-82ca-bbecd615988b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8df413e8-vxthq\" (UID: \"87a0b8bc-24a3-4450-82ca-bbecd615988b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq" Nov 03 15:29:57.667534 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.667500 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a0b8bc-24a3-4450-82ca-bbecd615988b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8df413e8-vxthq\" (UID: \"87a0b8bc-24a3-4450-82ca-bbecd615988b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq" Nov 03 15:29:57.667673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.667651 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbbpx\" (UniqueName: \"kubernetes.io/projected/87a0b8bc-24a3-4450-82ca-bbecd615988b-kube-api-access-gbbpx\") pod \"maas-api-revoke-sub-8df413e8-vxthq\" (UID: \"87a0b8bc-24a3-4450-82ca-bbecd615988b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq" Nov 03 15:29:57.769327 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.769229 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a0b8bc-24a3-4450-82ca-bbecd615988b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8df413e8-vxthq\" (UID: \"87a0b8bc-24a3-4450-82ca-bbecd615988b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq" Nov 03 15:29:57.769327 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.769309 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a0b8bc-24a3-4450-82ca-bbecd615988b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8df413e8-vxthq\" (UID: \"87a0b8bc-24a3-4450-82ca-bbecd615988b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq" Nov 03 15:29:57.769555 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.769385 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbbpx\" (UniqueName: \"kubernetes.io/projected/87a0b8bc-24a3-4450-82ca-bbecd615988b-kube-api-access-gbbpx\") pod \"maas-api-revoke-sub-8df413e8-vxthq\" (UID: \"87a0b8bc-24a3-4450-82ca-bbecd615988b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq" Nov 03 15:29:57.770097 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.770073 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a0b8bc-24a3-4450-82ca-bbecd615988b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8df413e8-vxthq\" (UID: \"87a0b8bc-24a3-4450-82ca-bbecd615988b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq" Nov 03 15:29:57.771920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.771893 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a0b8bc-24a3-4450-82ca-bbecd615988b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8df413e8-vxthq\" (UID: \"87a0b8bc-24a3-4450-82ca-bbecd615988b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq" Nov 03 15:29:57.830928 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.830899 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbbpx\" (UniqueName: \"kubernetes.io/projected/87a0b8bc-24a3-4450-82ca-bbecd615988b-kube-api-access-gbbpx\") pod \"maas-api-revoke-sub-8df413e8-vxthq\" (UID: \"87a0b8bc-24a3-4450-82ca-bbecd615988b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq" Nov 03 15:29:57.891460 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.891428 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq" Nov 03 15:29:57.918145 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:57.918119 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2" Nov 03 15:29:58.343919 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:29:58.343893 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a0b8bc_24a3_4450_82ca_bbecd615988b.slice/crio-14f26ab0cf5050801580bc687e356dfa3ae617667769d9442ef011cac30d48fd WatchSource:0}: Error finding container 14f26ab0cf5050801580bc687e356dfa3ae617667769d9442ef011cac30d48fd: Status 404 returned error can't find the container with id 14f26ab0cf5050801580bc687e356dfa3ae617667769d9442ef011cac30d48fd Nov 03 15:29:58.344857 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.344745 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq"] Nov 03 15:29:58.408413 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.408384 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62"] Nov 03 15:29:58.414242 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.414218 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62" Nov 03 15:29:58.447220 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.447185 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:58.447682 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.447658 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:58.448514 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.448493 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:29:58.474678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.474645 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62"] Nov 03 15:29:58.488944 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.488908 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mcxg\" (UniqueName: \"kubernetes.io/projected/76e93910-04b0-4f0e-99c1-f99b1b444b14-kube-api-access-6mcxg\") pod \"maas-api-revoke-sub-387e6deb-nrm62\" (UID: \"76e93910-04b0-4f0e-99c1-f99b1b444b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62" Nov 03 15:29:58.489217 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.489170 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e93910-04b0-4f0e-99c1-f99b1b444b14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-387e6deb-nrm62\" (UID: \"76e93910-04b0-4f0e-99c1-f99b1b444b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62" Nov 03 15:29:58.489537 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.489493 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e93910-04b0-4f0e-99c1-f99b1b444b14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-387e6deb-nrm62\" (UID: \"76e93910-04b0-4f0e-99c1-f99b1b444b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62" Nov 03 15:29:58.590905 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.590823 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mcxg\" (UniqueName: \"kubernetes.io/projected/76e93910-04b0-4f0e-99c1-f99b1b444b14-kube-api-access-6mcxg\") pod \"maas-api-revoke-sub-387e6deb-nrm62\" (UID: \"76e93910-04b0-4f0e-99c1-f99b1b444b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62" Nov 03 15:29:58.590905 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.590859 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e93910-04b0-4f0e-99c1-f99b1b444b14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-387e6deb-nrm62\" (UID: \"76e93910-04b0-4f0e-99c1-f99b1b444b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62" Nov 03 15:29:58.591124 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.590911 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e93910-04b0-4f0e-99c1-f99b1b444b14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-387e6deb-nrm62\" (UID: \"76e93910-04b0-4f0e-99c1-f99b1b444b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62" Nov 03 15:29:58.591556 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.591532 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e93910-04b0-4f0e-99c1-f99b1b444b14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-387e6deb-nrm62\" (UID: \"76e93910-04b0-4f0e-99c1-f99b1b444b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62" Nov 03 15:29:58.593149 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.593127 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e93910-04b0-4f0e-99c1-f99b1b444b14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-387e6deb-nrm62\" (UID: \"76e93910-04b0-4f0e-99c1-f99b1b444b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62" Nov 03 15:29:58.638834 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.638797 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:58.639233 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.639211 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:58.639958 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.639938 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:29:58.663362 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.663337 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mcxg\" (UniqueName: \"kubernetes.io/projected/76e93910-04b0-4f0e-99c1-f99b1b444b14-kube-api-access-6mcxg\") pod \"maas-api-revoke-sub-387e6deb-nrm62\" (UID: \"76e93910-04b0-4f0e-99c1-f99b1b444b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62" Nov 03 15:29:58.730945 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.730917 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62" Nov 03 15:29:58.927207 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.927127 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq" event={"ID":"87a0b8bc-24a3-4450-82ca-bbecd615988b","Type":"ContainerStarted","Data":"f422b883d10b7dcc6cdc11d84cb368316dbae43530890bc46040e7e42188b815"} Nov 03 15:29:58.927572 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:58.927216 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq" event={"ID":"87a0b8bc-24a3-4450-82ca-bbecd615988b","Type":"ContainerStarted","Data":"14f26ab0cf5050801580bc687e356dfa3ae617667769d9442ef011cac30d48fd"} Nov 03 15:29:59.045166 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:59.045117 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq" podStartSLOduration=2.04510366 podStartE2EDuration="2.04510366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:29:59.042393646 +0000 UTC m=+1308.885515211" watchObservedRunningTime="2026-11-03 15:29:59.04510366 +0000 UTC m=+1308.888225202" Nov 03 15:29:59.053502 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:59.053477 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn"] Nov 03 15:29:59.059793 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:59.059770 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-010f6743-kvqmn"] Nov 03 15:29:59.119392 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:59.119367 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62"] Nov 03 15:29:59.119565 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:29:59.119548 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e93910_04b0_4f0e_99c1_f99b1b444b14.slice/crio-60f68ee50449c9222e5cff000e770982ab46ccc58761213e6e224a2ba988ab44 WatchSource:0}: Error finding container 60f68ee50449c9222e5cff000e770982ab46ccc58761213e6e224a2ba988ab44: Status 404 returned error can't find the container with id 60f68ee50449c9222e5cff000e770982ab46ccc58761213e6e224a2ba988ab44 Nov 03 15:29:59.936389 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:59.936355 4557 generic.go:356] "Generic (PLEG): container finished" podID="76e93910-04b0-4f0e-99c1-f99b1b444b14" containerID="afe015b317288611d8a871d95a1144d082f4e633f78367ab975168c89d58a9b7" exitCode=0 Nov 03 15:29:59.936804 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:59.936435 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62" event={"ID":"76e93910-04b0-4f0e-99c1-f99b1b444b14","Type":"ContainerDied","Data":"afe015b317288611d8a871d95a1144d082f4e633f78367ab975168c89d58a9b7"} Nov 03 15:29:59.936804 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:59.936486 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62" event={"ID":"76e93910-04b0-4f0e-99c1-f99b1b444b14","Type":"ContainerStarted","Data":"60f68ee50449c9222e5cff000e770982ab46ccc58761213e6e224a2ba988ab44"} Nov 03 15:29:59.938339 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:59.938316 4557 generic.go:356] "Generic (PLEG): container finished" podID="87a0b8bc-24a3-4450-82ca-bbecd615988b" containerID="f422b883d10b7dcc6cdc11d84cb368316dbae43530890bc46040e7e42188b815" exitCode=0 Nov 03 15:29:59.938446 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:29:59.938395 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq" event={"ID":"87a0b8bc-24a3-4450-82ca-bbecd615988b","Type":"ContainerDied","Data":"f422b883d10b7dcc6cdc11d84cb368316dbae43530890bc46040e7e42188b815"} Nov 03 15:30:00.225607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.225514 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx"] Nov 03 15:30:00.230385 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.230367 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9"] Nov 03 15:30:00.230491 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.230473 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" Nov 03 15:30:00.234323 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.234306 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" Nov 03 15:30:00.251881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.251861 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg"] Nov 03 15:30:00.256517 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.256494 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" Nov 03 15:30:00.287693 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.287660 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5165cdf-638c-4399-b2db-6526fa123371" path="/var/lib/kubelet/pods/a5165cdf-638c-4399-b2db-6526fa123371/volumes" Nov 03 15:30:00.300029 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.299999 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx"] Nov 03 15:30:00.310443 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.310414 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpnhm\" (UniqueName: \"kubernetes.io/projected/be6660d4-9d6b-4706-9cfd-0d204e1b4e45-kube-api-access-jpnhm\") pod \"maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx\" (UID: \"be6660d4-9d6b-4706-9cfd-0d204e1b4e45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" Nov 03 15:30:00.310621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.310551 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6660d4-9d6b-4706-9cfd-0d204e1b4e45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx\" (UID: \"be6660d4-9d6b-4706-9cfd-0d204e1b4e45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" Nov 03 15:30:00.310704 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.310684 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whw7h\" (UniqueName: \"kubernetes.io/projected/192929b4-7ed3-4e9d-9a32-f18088dc5910-kube-api-access-whw7h\") pod \"maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg\" (UID: \"192929b4-7ed3-4e9d-9a32-f18088dc5910\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" Nov 03 15:30:00.310811 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.310789 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/192929b4-7ed3-4e9d-9a32-f18088dc5910-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg\" (UID: \"192929b4-7ed3-4e9d-9a32-f18088dc5910\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" Nov 03 15:30:00.310902 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.310878 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w5n9\" (UniqueName: \"kubernetes.io/projected/c7b8f36c-0d65-4523-b767-cd6970eb594e-kube-api-access-9w5n9\") pod \"maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9\" (UID: \"c7b8f36c-0d65-4523-b767-cd6970eb594e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" Nov 03 15:30:00.311016 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.310999 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b8f36c-0d65-4523-b767-cd6970eb594e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9\" (UID: \"c7b8f36c-0d65-4523-b767-cd6970eb594e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" Nov 03 15:30:00.354145 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.354113 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9"] Nov 03 15:30:00.358338 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.358311 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg"] Nov 03 15:30:00.412019 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.411975 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpnhm\" (UniqueName: \"kubernetes.io/projected/be6660d4-9d6b-4706-9cfd-0d204e1b4e45-kube-api-access-jpnhm\") pod \"maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx\" (UID: \"be6660d4-9d6b-4706-9cfd-0d204e1b4e45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" Nov 03 15:30:00.412187 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.412039 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6660d4-9d6b-4706-9cfd-0d204e1b4e45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx\" (UID: \"be6660d4-9d6b-4706-9cfd-0d204e1b4e45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" Nov 03 15:30:00.412187 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.412066 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whw7h\" (UniqueName: \"kubernetes.io/projected/192929b4-7ed3-4e9d-9a32-f18088dc5910-kube-api-access-whw7h\") pod \"maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg\" (UID: \"192929b4-7ed3-4e9d-9a32-f18088dc5910\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" Nov 03 15:30:00.412187 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.412100 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/192929b4-7ed3-4e9d-9a32-f18088dc5910-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg\" (UID: \"192929b4-7ed3-4e9d-9a32-f18088dc5910\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" Nov 03 15:30:00.412187 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.412132 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w5n9\" (UniqueName: \"kubernetes.io/projected/c7b8f36c-0d65-4523-b767-cd6970eb594e-kube-api-access-9w5n9\") pod \"maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9\" (UID: \"c7b8f36c-0d65-4523-b767-cd6970eb594e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" Nov 03 15:30:00.412409 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.412214 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b8f36c-0d65-4523-b767-cd6970eb594e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9\" (UID: \"c7b8f36c-0d65-4523-b767-cd6970eb594e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" Nov 03 15:30:00.412863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.412843 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6660d4-9d6b-4706-9cfd-0d204e1b4e45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx\" (UID: \"be6660d4-9d6b-4706-9cfd-0d204e1b4e45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" Nov 03 15:30:00.412936 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.412913 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/192929b4-7ed3-4e9d-9a32-f18088dc5910-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg\" (UID: \"192929b4-7ed3-4e9d-9a32-f18088dc5910\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" Nov 03 15:30:00.413016 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.413000 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b8f36c-0d65-4523-b767-cd6970eb594e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9\" (UID: \"c7b8f36c-0d65-4523-b767-cd6970eb594e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" Nov 03 15:30:00.473603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.473552 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w5n9\" (UniqueName: \"kubernetes.io/projected/c7b8f36c-0d65-4523-b767-cd6970eb594e-kube-api-access-9w5n9\") pod \"maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9\" (UID: \"c7b8f36c-0d65-4523-b767-cd6970eb594e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" Nov 03 15:30:00.473736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.473622 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpnhm\" (UniqueName: \"kubernetes.io/projected/be6660d4-9d6b-4706-9cfd-0d204e1b4e45-kube-api-access-jpnhm\") pod \"maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx\" (UID: \"be6660d4-9d6b-4706-9cfd-0d204e1b4e45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" Nov 03 15:30:00.473736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.473666 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whw7h\" (UniqueName: \"kubernetes.io/projected/192929b4-7ed3-4e9d-9a32-f18088dc5910-kube-api-access-whw7h\") pod \"maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg\" (UID: \"192929b4-7ed3-4e9d-9a32-f18088dc5910\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" Nov 03 15:30:00.526151 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.526071 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:00.545169 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.545128 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" Nov 03 15:30:00.551546 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.551514 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" Nov 03 15:30:00.569611 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.569403 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" Nov 03 15:30:00.941239 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.941209 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx"] Nov 03 15:30:00.996540 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:00.996508 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9"] Nov 03 15:30:00.997726 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:30:00.997697 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b8f36c_0d65_4523_b767_cd6970eb594e.slice/crio-cd991d968e7da45065906e6006b5fa3f421381722a560d1d87fb11474559503f WatchSource:0}: Error finding container cd991d968e7da45065906e6006b5fa3f421381722a560d1d87fb11474559503f: Status 404 returned error can't find the container with id cd991d968e7da45065906e6006b5fa3f421381722a560d1d87fb11474559503f Nov 03 15:30:01.045784 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.045756 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg"] Nov 03 15:30:01.051156 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:30:01.051118 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192929b4_7ed3_4e9d_9a32_f18088dc5910.slice/crio-e338e868a90e9ded1796bd8bd214c8dc60dc35eb77f055e05b3cddbabf8769a8 WatchSource:0}: Error finding container e338e868a90e9ded1796bd8bd214c8dc60dc35eb77f055e05b3cddbabf8769a8: Status 404 returned error can't find the container with id e338e868a90e9ded1796bd8bd214c8dc60dc35eb77f055e05b3cddbabf8769a8 Nov 03 15:30:01.188604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.188513 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq" Nov 03 15:30:01.190565 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.190545 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62" Nov 03 15:30:01.224138 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.224104 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76e93910-04b0-4f0e-99c1-f99b1b444b14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e93910-04b0-4f0e-99c1-f99b1b444b14-maas-api-xks-ca\") pod \"76e93910-04b0-4f0e-99c1-f99b1b444b14\" (UID: \"76e93910-04b0-4f0e-99c1-f99b1b444b14\") " Nov 03 15:30:01.224289 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.224155 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76e93910-04b0-4f0e-99c1-f99b1b444b14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e93910-04b0-4f0e-99c1-f99b1b444b14-maas-api-openshift-ca\") pod \"76e93910-04b0-4f0e-99c1-f99b1b444b14\" (UID: \"76e93910-04b0-4f0e-99c1-f99b1b444b14\") " Nov 03 15:30:01.224289 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.224270 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a0b8bc-24a3-4450-82ca-bbecd615988b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a0b8bc-24a3-4450-82ca-bbecd615988b-maas-api-openshift-ca\") pod \"87a0b8bc-24a3-4450-82ca-bbecd615988b\" (UID: \"87a0b8bc-24a3-4450-82ca-bbecd615988b\") " Nov 03 15:30:01.224401 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.224347 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e93910-04b0-4f0e-99c1-f99b1b444b14-kube-api-access-6mcxg\" (UniqueName: \"kubernetes.io/projected/76e93910-04b0-4f0e-99c1-f99b1b444b14-kube-api-access-6mcxg\") pod \"76e93910-04b0-4f0e-99c1-f99b1b444b14\" (UID: \"76e93910-04b0-4f0e-99c1-f99b1b444b14\") " Nov 03 15:30:01.224401 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.224382 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a0b8bc-24a3-4450-82ca-bbecd615988b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a0b8bc-24a3-4450-82ca-bbecd615988b-maas-api-xks-ca\") pod \"87a0b8bc-24a3-4450-82ca-bbecd615988b\" (UID: \"87a0b8bc-24a3-4450-82ca-bbecd615988b\") " Nov 03 15:30:01.224505 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.224410 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a0b8bc-24a3-4450-82ca-bbecd615988b-kube-api-access-gbbpx\" (UniqueName: \"kubernetes.io/projected/87a0b8bc-24a3-4450-82ca-bbecd615988b-kube-api-access-gbbpx\") pod \"87a0b8bc-24a3-4450-82ca-bbecd615988b\" (UID: \"87a0b8bc-24a3-4450-82ca-bbecd615988b\") " Nov 03 15:30:01.224559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.224539 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76e93910-04b0-4f0e-99c1-f99b1b444b14-maas-api-openshift-ca" pod "76e93910-04b0-4f0e-99c1-f99b1b444b14" (UID: "76e93910-04b0-4f0e-99c1-f99b1b444b14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:01.224662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.224638 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a0b8bc-24a3-4450-82ca-bbecd615988b-maas-api-openshift-ca" pod "87a0b8bc-24a3-4450-82ca-bbecd615988b" (UID: "87a0b8bc-24a3-4450-82ca-bbecd615988b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:01.225037 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.225015 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a0b8bc-24a3-4450-82ca-bbecd615988b-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:01.225134 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.225041 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e93910-04b0-4f0e-99c1-f99b1b444b14-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:01.226718 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.226676 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76e93910-04b0-4f0e-99c1-f99b1b444b14-maas-api-xks-ca" pod "76e93910-04b0-4f0e-99c1-f99b1b444b14" (UID: "76e93910-04b0-4f0e-99c1-f99b1b444b14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:30:01.227073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.227045 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a0b8bc-24a3-4450-82ca-bbecd615988b-maas-api-xks-ca" pod "87a0b8bc-24a3-4450-82ca-bbecd615988b" (UID: "87a0b8bc-24a3-4450-82ca-bbecd615988b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:30:01.227178 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.227125 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e93910-04b0-4f0e-99c1-f99b1b444b14-kube-api-access-6mcxg" pod "76e93910-04b0-4f0e-99c1-f99b1b444b14" (UID: "76e93910-04b0-4f0e-99c1-f99b1b444b14"). InnerVolumeSpecName "kube-api-access-6mcxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:01.227235 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.227186 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a0b8bc-24a3-4450-82ca-bbecd615988b-kube-api-access-gbbpx" pod "87a0b8bc-24a3-4450-82ca-bbecd615988b" (UID: "87a0b8bc-24a3-4450-82ca-bbecd615988b"). InnerVolumeSpecName "kube-api-access-gbbpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:01.326470 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.326434 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mcxg\" (UniqueName: \"kubernetes.io/projected/76e93910-04b0-4f0e-99c1-f99b1b444b14-kube-api-access-6mcxg\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:01.326470 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.326466 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a0b8bc-24a3-4450-82ca-bbecd615988b-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:01.326470 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.326476 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbbpx\" (UniqueName: \"kubernetes.io/projected/87a0b8bc-24a3-4450-82ca-bbecd615988b-kube-api-access-gbbpx\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:01.326783 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.326486 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e93910-04b0-4f0e-99c1-f99b1b444b14-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:01.501106 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.501079 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf"] Nov 03 15:30:01.501633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.501614 4557 cpu_manager.go:414] "containerMap: removing container" podUID="87a0b8bc-24a3-4450-82ca-bbecd615988b" containerName="revoke-keys" Nov 03 15:30:01.501633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.501631 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a0b8bc-24a3-4450-82ca-bbecd615988b" containerName="revoke-keys" Nov 03 15:30:01.501783 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.501659 4557 cpu_manager.go:414] "containerMap: removing container" podUID="76e93910-04b0-4f0e-99c1-f99b1b444b14" containerName="revoke-keys" Nov 03 15:30:01.501783 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.501665 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e93910-04b0-4f0e-99c1-f99b1b444b14" containerName="revoke-keys" Nov 03 15:30:01.501783 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.501755 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a0b8bc-24a3-4450-82ca-bbecd615988b" containerName="revoke-keys" Nov 03 15:30:01.501783 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.501765 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e93910-04b0-4f0e-99c1-f99b1b444b14" containerName="revoke-keys" Nov 03 15:30:01.505059 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.505041 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.567300 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.567267 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf"] Nov 03 15:30:01.629818 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.629785 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.629965 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.629860 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.629965 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.629933 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.630067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.629965 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.630067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.629984 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.630167 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.630123 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.630219 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.630164 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.630219 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.630183 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-694j6\" (UniqueName: \"kubernetes.io/projected/82f7013a-125c-4b34-8f4b-b72f312179d8-kube-api-access-694j6\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.630320 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.630307 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f7013a-125c-4b34-8f4b-b72f312179d8-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.731846 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.731800 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.732016 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.731876 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.732016 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.731904 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-694j6\" (UniqueName: \"kubernetes.io/projected/82f7013a-125c-4b34-8f4b-b72f312179d8-kube-api-access-694j6\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.732016 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.731983 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f7013a-125c-4b34-8f4b-b72f312179d8-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.732199 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.732040 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.732199 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.732149 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.732301 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.732203 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.732301 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.732238 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.732301 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.732268 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.732301 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.732269 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.732647 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.732595 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.732934 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.732900 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.733438 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.733415 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.733713 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.733688 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f7013a-125c-4b34-8f4b-b72f312179d8-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.736102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.736048 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.736292 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.736276 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.793810 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.793782 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.793973 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.793954 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-694j6\" (UniqueName: \"kubernetes.io/projected/82f7013a-125c-4b34-8f4b-b72f312179d8-kube-api-access-694j6\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.819429 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.819393 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:01.957402 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.957373 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq" Nov 03 15:30:01.957916 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.957397 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq" event={"ID":"87a0b8bc-24a3-4450-82ca-bbecd615988b","Type":"ContainerDied","Data":"14f26ab0cf5050801580bc687e356dfa3ae617667769d9442ef011cac30d48fd"} Nov 03 15:30:01.957916 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.957450 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14f26ab0cf5050801580bc687e356dfa3ae617667769d9442ef011cac30d48fd" Nov 03 15:30:01.959390 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.959366 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" event={"ID":"192929b4-7ed3-4e9d-9a32-f18088dc5910","Type":"ContainerStarted","Data":"956a151ea5e3a13c84d4b738ef2b43c7444b848fdb4b2f97642f5edcb5191d9a"} Nov 03 15:30:01.959530 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.959416 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" event={"ID":"192929b4-7ed3-4e9d-9a32-f18088dc5910","Type":"ContainerStarted","Data":"e338e868a90e9ded1796bd8bd214c8dc60dc35eb77f055e05b3cddbabf8769a8"} Nov 03 15:30:01.961364 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.961342 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" event={"ID":"c7b8f36c-0d65-4523-b767-cd6970eb594e","Type":"ContainerStarted","Data":"aea3f65cd07476f91b3cae2cdc6821915a1f0240d493bc9d549dbe8c59a36a97"} Nov 03 15:30:01.961468 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.961379 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" event={"ID":"c7b8f36c-0d65-4523-b767-cd6970eb594e","Type":"ContainerStarted","Data":"cd991d968e7da45065906e6006b5fa3f421381722a560d1d87fb11474559503f"} Nov 03 15:30:01.962837 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.962817 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62" Nov 03 15:30:01.962961 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.962821 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62" event={"ID":"76e93910-04b0-4f0e-99c1-f99b1b444b14","Type":"ContainerDied","Data":"60f68ee50449c9222e5cff000e770982ab46ccc58761213e6e224a2ba988ab44"} Nov 03 15:30:01.962961 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.962932 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60f68ee50449c9222e5cff000e770982ab46ccc58761213e6e224a2ba988ab44" Nov 03 15:30:01.964683 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.964659 4557 generic.go:356] "Generic (PLEG): container finished" podID="be6660d4-9d6b-4706-9cfd-0d204e1b4e45" containerID="2f8f00600bd6d628f35d1ea3bca50731d711fa8abc7892b8d332c2d53fac0d35" exitCode=60 Nov 03 15:30:01.964775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.964739 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" event={"ID":"be6660d4-9d6b-4706-9cfd-0d204e1b4e45","Type":"ContainerDied","Data":"2f8f00600bd6d628f35d1ea3bca50731d711fa8abc7892b8d332c2d53fac0d35"} Nov 03 15:30:01.964836 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.964786 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" event={"ID":"be6660d4-9d6b-4706-9cfd-0d204e1b4e45","Type":"ContainerStarted","Data":"43ce364767919e715915e7978463fd00ed51dadc70daf10dbdf8c0b26fec6a75"} Nov 03 15:30:01.964923 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:01.964906 4557 scope.go:122] "RemoveContainer" containerID="2f8f00600bd6d628f35d1ea3bca50731d711fa8abc7892b8d332c2d53fac0d35" Nov 03 15:30:02.050092 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:02.050053 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6"] Nov 03 15:30:02.054350 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:02.054324 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49de6b74-9l6d6"] Nov 03 15:30:02.076748 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:02.076677 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" podStartSLOduration=2.0766545 podStartE2EDuration="2.0766545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:02.073681365 +0000 UTC m=+1311.916802908" watchObservedRunningTime="2026-11-03 15:30:02.0766545 +0000 UTC m=+1311.919776055" Nov 03 15:30:02.270553 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:02.270521 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf"] Nov 03 15:30:02.270677 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:30:02.270614 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f7013a_125c_4b34_8f4b_b72f312179d8.slice/crio-c62a2440c9f433888d2bc75736dca9ec3521d2568c0e703cab5cf1f2accefe3b WatchSource:0}: Error finding container c62a2440c9f433888d2bc75736dca9ec3521d2568c0e703cab5cf1f2accefe3b: Status 404 returned error can't find the container with id c62a2440c9f433888d2bc75736dca9ec3521d2568c0e703cab5cf1f2accefe3b Nov 03 15:30:02.273151 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:02.273121 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:02.273248 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:02.273196 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:02.273306 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:02.273243 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:02.288418 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:02.288388 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b715d1-2833-48d1-a1fc-e30c1abcf0ab" path="/var/lib/kubelet/pods/f1b715d1-2833-48d1-a1fc-e30c1abcf0ab/volumes" Nov 03 15:30:02.373659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:02.373606 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" podStartSLOduration=2.373566553 podStartE2EDuration="2.373566553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:02.36972121 +0000 UTC m=+1312.212842753" watchObservedRunningTime="2026-11-03 15:30:02.373566553 +0000 UTC m=+1312.216688149" Nov 03 15:30:02.639717 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:02.639679 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:02.962350 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:02.962321 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5"] Nov 03 15:30:02.966683 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:02.966662 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5" Nov 03 15:30:02.982603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:02.976739 4557 generic.go:356] "Generic (PLEG): container finished" podID="192929b4-7ed3-4e9d-9a32-f18088dc5910" containerID="956a151ea5e3a13c84d4b738ef2b43c7444b848fdb4b2f97642f5edcb5191d9a" exitCode=60 Nov 03 15:30:02.982603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:02.976894 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" event={"ID":"192929b4-7ed3-4e9d-9a32-f18088dc5910","Type":"ContainerDied","Data":"956a151ea5e3a13c84d4b738ef2b43c7444b848fdb4b2f97642f5edcb5191d9a"} Nov 03 15:30:02.982603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:02.979092 4557 scope.go:122] "RemoveContainer" containerID="956a151ea5e3a13c84d4b738ef2b43c7444b848fdb4b2f97642f5edcb5191d9a" Nov 03 15:30:03.010871 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.010841 4557 generic.go:356] "Generic (PLEG): container finished" podID="c7b8f36c-0d65-4523-b767-cd6970eb594e" containerID="aea3f65cd07476f91b3cae2cdc6821915a1f0240d493bc9d549dbe8c59a36a97" exitCode=60 Nov 03 15:30:03.010999 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.010969 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" event={"ID":"c7b8f36c-0d65-4523-b767-cd6970eb594e","Type":"ContainerDied","Data":"aea3f65cd07476f91b3cae2cdc6821915a1f0240d493bc9d549dbe8c59a36a97"} Nov 03 15:30:03.011278 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.011261 4557 scope.go:122] "RemoveContainer" containerID="aea3f65cd07476f91b3cae2cdc6821915a1f0240d493bc9d549dbe8c59a36a97" Nov 03 15:30:03.019601 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.019566 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:30:03.024271 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.024241 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" event={"ID":"be6660d4-9d6b-4706-9cfd-0d204e1b4e45","Type":"ContainerStarted","Data":"2a49f9870306216db8faf03288642bc88d9cb3ace38557394b84b7ef71ab22ab"} Nov 03 15:30:03.025384 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.025364 4557 scope.go:122] "RemoveContainer" containerID="2a49f9870306216db8faf03288642bc88d9cb3ace38557394b84b7ef71ab22ab" Nov 03 15:30:03.025564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.025544 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5"] Nov 03 15:30:03.025752 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:30:03.025726 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx_odh-ai-gateway-infra(be6660d4-9d6b-4706-9cfd-0d204e1b4e45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" podUID="be6660d4-9d6b-4706-9cfd-0d204e1b4e45" Nov 03 15:30:03.028939 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.027203 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" event={"ID":"82f7013a-125c-4b34-8f4b-b72f312179d8","Type":"ContainerStarted","Data":"292552c5af7a71dc650cb5e0c75fbc52d65fd3973a30aeefe7a3c8601beaa1bd"} Nov 03 15:30:03.028939 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.027244 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" event={"ID":"82f7013a-125c-4b34-8f4b-b72f312179d8","Type":"ContainerStarted","Data":"c62a2440c9f433888d2bc75736dca9ec3521d2568c0e703cab5cf1f2accefe3b"} Nov 03 15:30:03.049681 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.049655 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/759a1535-c79d-4518-ad7f-4219c9d28059-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cab8b4ee-9h8w5\" (UID: \"759a1535-c79d-4518-ad7f-4219c9d28059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5" Nov 03 15:30:03.049788 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.049752 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/759a1535-c79d-4518-ad7f-4219c9d28059-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cab8b4ee-9h8w5\" (UID: \"759a1535-c79d-4518-ad7f-4219c9d28059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5" Nov 03 15:30:03.049906 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.049890 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp4mc\" (UniqueName: \"kubernetes.io/projected/759a1535-c79d-4518-ad7f-4219c9d28059-kube-api-access-sp4mc\") pod \"maas-api-revoke-sub-cab8b4ee-9h8w5\" (UID: \"759a1535-c79d-4518-ad7f-4219c9d28059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5" Nov 03 15:30:03.151209 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.151166 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp4mc\" (UniqueName: \"kubernetes.io/projected/759a1535-c79d-4518-ad7f-4219c9d28059-kube-api-access-sp4mc\") pod \"maas-api-revoke-sub-cab8b4ee-9h8w5\" (UID: \"759a1535-c79d-4518-ad7f-4219c9d28059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5" Nov 03 15:30:03.151646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.151622 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/759a1535-c79d-4518-ad7f-4219c9d28059-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cab8b4ee-9h8w5\" (UID: \"759a1535-c79d-4518-ad7f-4219c9d28059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5" Nov 03 15:30:03.151760 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.151710 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/759a1535-c79d-4518-ad7f-4219c9d28059-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cab8b4ee-9h8w5\" (UID: \"759a1535-c79d-4518-ad7f-4219c9d28059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5" Nov 03 15:30:03.152301 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.152277 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/759a1535-c79d-4518-ad7f-4219c9d28059-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cab8b4ee-9h8w5\" (UID: \"759a1535-c79d-4518-ad7f-4219c9d28059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5" Nov 03 15:30:03.154030 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.154011 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/759a1535-c79d-4518-ad7f-4219c9d28059-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cab8b4ee-9h8w5\" (UID: \"759a1535-c79d-4518-ad7f-4219c9d28059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5" Nov 03 15:30:03.217782 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.217710 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp4mc\" (UniqueName: \"kubernetes.io/projected/759a1535-c79d-4518-ad7f-4219c9d28059-kube-api-access-sp4mc\") pod \"maas-api-revoke-sub-cab8b4ee-9h8w5\" (UID: \"759a1535-c79d-4518-ad7f-4219c9d28059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5" Nov 03 15:30:03.320539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.320512 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5" Nov 03 15:30:03.454804 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.454737 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" podStartSLOduration=2.454722404 podStartE2EDuration="2.454722404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:03.451354457 +0000 UTC m=+1313.294476029" watchObservedRunningTime="2026-11-03 15:30:03.454722404 +0000 UTC m=+1313.297843946" Nov 03 15:30:03.719734 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:30:03.719699 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759a1535_c79d_4518_ad7f_4219c9d28059.slice/crio-509e3c9adc7f051fc64cc2f2632ccd2e07560fcd300a43f568e643c49668f994 WatchSource:0}: Error finding container 509e3c9adc7f051fc64cc2f2632ccd2e07560fcd300a43f568e643c49668f994: Status 404 returned error can't find the container with id 509e3c9adc7f051fc64cc2f2632ccd2e07560fcd300a43f568e643c49668f994 Nov 03 15:30:03.720017 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.719994 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5"] Nov 03 15:30:03.819839 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.819799 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:03.821632 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.821604 4557 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 15:30:03.821742 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:03.821652 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" podUID="82f7013a-125c-4b34-8f4b-b72f312179d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 15:30:04.035892 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:04.035849 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" event={"ID":"192929b4-7ed3-4e9d-9a32-f18088dc5910","Type":"ContainerStarted","Data":"e82c32cecdcc83b9c4d3d655403fe8116fb0305bcccd9e24ab01d30fb2febf07"} Nov 03 15:30:04.037943 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:04.037916 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" event={"ID":"c7b8f36c-0d65-4523-b767-cd6970eb594e","Type":"ContainerStarted","Data":"b7256769e53da54b80c62f72af2adf06e70690a32a26013e9dda1313e8b37e73"} Nov 03 15:30:04.039733 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:04.039706 4557 generic.go:356] "Generic (PLEG): container finished" podID="be6660d4-9d6b-4706-9cfd-0d204e1b4e45" containerID="2a49f9870306216db8faf03288642bc88d9cb3ace38557394b84b7ef71ab22ab" exitCode=60 Nov 03 15:30:04.039858 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:04.039781 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" event={"ID":"be6660d4-9d6b-4706-9cfd-0d204e1b4e45","Type":"ContainerDied","Data":"2a49f9870306216db8faf03288642bc88d9cb3ace38557394b84b7ef71ab22ab"} Nov 03 15:30:04.039923 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:04.039857 4557 scope.go:122] "RemoveContainer" containerID="2f8f00600bd6d628f35d1ea3bca50731d711fa8abc7892b8d332c2d53fac0d35" Nov 03 15:30:04.040005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:04.039984 4557 scope.go:122] "RemoveContainer" containerID="2a49f9870306216db8faf03288642bc88d9cb3ace38557394b84b7ef71ab22ab" Nov 03 15:30:04.040240 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:30:04.040223 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx_odh-ai-gateway-infra(be6660d4-9d6b-4706-9cfd-0d204e1b4e45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" podUID="be6660d4-9d6b-4706-9cfd-0d204e1b4e45" Nov 03 15:30:04.041615 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:04.041562 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5" event={"ID":"759a1535-c79d-4518-ad7f-4219c9d28059","Type":"ContainerStarted","Data":"509e3c9adc7f051fc64cc2f2632ccd2e07560fcd300a43f568e643c49668f994"} Nov 03 15:30:04.819795 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:04.819758 4557 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 15:30:04.819989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:04.819818 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" podUID="82f7013a-125c-4b34-8f4b-b72f312179d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 15:30:05.052769 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:05.052735 4557 generic.go:356] "Generic (PLEG): container finished" podID="192929b4-7ed3-4e9d-9a32-f18088dc5910" containerID="e82c32cecdcc83b9c4d3d655403fe8116fb0305bcccd9e24ab01d30fb2febf07" exitCode=60 Nov 03 15:30:05.053199 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:05.052832 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" event={"ID":"192929b4-7ed3-4e9d-9a32-f18088dc5910","Type":"ContainerDied","Data":"e82c32cecdcc83b9c4d3d655403fe8116fb0305bcccd9e24ab01d30fb2febf07"} Nov 03 15:30:05.053199 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:05.052910 4557 scope.go:122] "RemoveContainer" containerID="956a151ea5e3a13c84d4b738ef2b43c7444b848fdb4b2f97642f5edcb5191d9a" Nov 03 15:30:05.053199 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:05.053093 4557 scope.go:122] "RemoveContainer" containerID="e82c32cecdcc83b9c4d3d655403fe8116fb0305bcccd9e24ab01d30fb2febf07" Nov 03 15:30:05.053454 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:30:05.053431 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg_odh-ai-gateway-infra(192929b4-7ed3-4e9d-9a32-f18088dc5910)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" podUID="192929b4-7ed3-4e9d-9a32-f18088dc5910" Nov 03 15:30:05.055118 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:05.055098 4557 generic.go:356] "Generic (PLEG): container finished" podID="c7b8f36c-0d65-4523-b767-cd6970eb594e" containerID="b7256769e53da54b80c62f72af2adf06e70690a32a26013e9dda1313e8b37e73" exitCode=60 Nov 03 15:30:05.055204 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:05.055178 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" event={"ID":"c7b8f36c-0d65-4523-b767-cd6970eb594e","Type":"ContainerDied","Data":"b7256769e53da54b80c62f72af2adf06e70690a32a26013e9dda1313e8b37e73"} Nov 03 15:30:05.055429 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:05.055413 4557 scope.go:122] "RemoveContainer" containerID="b7256769e53da54b80c62f72af2adf06e70690a32a26013e9dda1313e8b37e73" Nov 03 15:30:05.055781 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:30:05.055756 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9_odh-ai-gateway-infra(c7b8f36c-0d65-4523-b767-cd6970eb594e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" podUID="c7b8f36c-0d65-4523-b767-cd6970eb594e" Nov 03 15:30:05.057101 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:05.057085 4557 scope.go:122] "RemoveContainer" containerID="2a49f9870306216db8faf03288642bc88d9cb3ace38557394b84b7ef71ab22ab" Nov 03 15:30:05.057362 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:30:05.057342 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx_odh-ai-gateway-infra(be6660d4-9d6b-4706-9cfd-0d204e1b4e45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" podUID="be6660d4-9d6b-4706-9cfd-0d204e1b4e45" Nov 03 15:30:05.058313 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:05.058293 4557 generic.go:356] "Generic (PLEG): container finished" podID="759a1535-c79d-4518-ad7f-4219c9d28059" containerID="73c294e7aefa6bc62ec893ad3f1f8bf1b9ebc1a74ed26a4a79d561c2632eb6c2" exitCode=0 Nov 03 15:30:05.058392 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:05.058356 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5" event={"ID":"759a1535-c79d-4518-ad7f-4219c9d28059","Type":"ContainerDied","Data":"73c294e7aefa6bc62ec893ad3f1f8bf1b9ebc1a74ed26a4a79d561c2632eb6c2"} Nov 03 15:30:05.071609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:05.071574 4557 scope.go:122] "RemoveContainer" containerID="aea3f65cd07476f91b3cae2cdc6821915a1f0240d493bc9d549dbe8c59a36a97" Nov 03 15:30:05.109332 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:05.109301 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5"] Nov 03 15:30:05.113631 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:05.113571 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b7c154-26fg5"] Nov 03 15:30:05.613813 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:05.604105 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:05.820658 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:05.820623 4557 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 15:30:05.820820 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:05.820687 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" podUID="82f7013a-125c-4b34-8f4b-b72f312179d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 15:30:06.068925 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:06.068892 4557 scope.go:122] "RemoveContainer" containerID="e82c32cecdcc83b9c4d3d655403fe8116fb0305bcccd9e24ab01d30fb2febf07" Nov 03 15:30:06.069346 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:30:06.069244 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg_odh-ai-gateway-infra(192929b4-7ed3-4e9d-9a32-f18088dc5910)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" podUID="192929b4-7ed3-4e9d-9a32-f18088dc5910" Nov 03 15:30:06.070518 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:06.070495 4557 scope.go:122] "RemoveContainer" containerID="b7256769e53da54b80c62f72af2adf06e70690a32a26013e9dda1313e8b37e73" Nov 03 15:30:06.070972 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:30:06.070839 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9_odh-ai-gateway-infra(c7b8f36c-0d65-4523-b767-cd6970eb594e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" podUID="c7b8f36c-0d65-4523-b767-cd6970eb594e" Nov 03 15:30:06.264699 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:06.264675 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5" Nov 03 15:30:06.310819 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:06.310784 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c8ee92-ba14-4954-a3fd-fc2e492b8a05" path="/var/lib/kubelet/pods/43c8ee92-ba14-4954-a3fd-fc2e492b8a05/volumes" Nov 03 15:30:06.422445 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:06.422360 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/759a1535-c79d-4518-ad7f-4219c9d28059-kube-api-access-sp4mc\" (UniqueName: \"kubernetes.io/projected/759a1535-c79d-4518-ad7f-4219c9d28059-kube-api-access-sp4mc\") pod \"759a1535-c79d-4518-ad7f-4219c9d28059\" (UID: \"759a1535-c79d-4518-ad7f-4219c9d28059\") " Nov 03 15:30:06.422618 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:06.422463 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/759a1535-c79d-4518-ad7f-4219c9d28059-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/759a1535-c79d-4518-ad7f-4219c9d28059-maas-api-openshift-ca\") pod \"759a1535-c79d-4518-ad7f-4219c9d28059\" (UID: \"759a1535-c79d-4518-ad7f-4219c9d28059\") " Nov 03 15:30:06.422618 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:06.422517 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/759a1535-c79d-4518-ad7f-4219c9d28059-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/759a1535-c79d-4518-ad7f-4219c9d28059-maas-api-xks-ca\") pod \"759a1535-c79d-4518-ad7f-4219c9d28059\" (UID: \"759a1535-c79d-4518-ad7f-4219c9d28059\") " Nov 03 15:30:06.422895 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:06.422869 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/759a1535-c79d-4518-ad7f-4219c9d28059-maas-api-openshift-ca" pod "759a1535-c79d-4518-ad7f-4219c9d28059" (UID: "759a1535-c79d-4518-ad7f-4219c9d28059"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:06.424537 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:06.424513 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/759a1535-c79d-4518-ad7f-4219c9d28059-maas-api-xks-ca" pod "759a1535-c79d-4518-ad7f-4219c9d28059" (UID: "759a1535-c79d-4518-ad7f-4219c9d28059"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:30:06.424650 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:06.424615 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759a1535-c79d-4518-ad7f-4219c9d28059-kube-api-access-sp4mc" pod "759a1535-c79d-4518-ad7f-4219c9d28059" (UID: "759a1535-c79d-4518-ad7f-4219c9d28059"). InnerVolumeSpecName "kube-api-access-sp4mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:06.524092 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:06.524059 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp4mc\" (UniqueName: \"kubernetes.io/projected/759a1535-c79d-4518-ad7f-4219c9d28059-kube-api-access-sp4mc\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:06.524092 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:06.524089 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/759a1535-c79d-4518-ad7f-4219c9d28059-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:06.524298 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:06.524101 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/759a1535-c79d-4518-ad7f-4219c9d28059-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:06.820156 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:06.820118 4557 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 15:30:06.820331 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:06.820175 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" podUID="82f7013a-125c-4b34-8f4b-b72f312179d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 15:30:07.103309 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.103221 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5" event={"ID":"759a1535-c79d-4518-ad7f-4219c9d28059","Type":"ContainerDied","Data":"509e3c9adc7f051fc64cc2f2632ccd2e07560fcd300a43f568e643c49668f994"} Nov 03 15:30:07.103309 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.103280 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="509e3c9adc7f051fc64cc2f2632ccd2e07560fcd300a43f568e643c49668f994" Nov 03 15:30:07.103309 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.103228 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5" Nov 03 15:30:07.354342 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.354263 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6"] Nov 03 15:30:07.354793 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.354780 4557 cpu_manager.go:414] "containerMap: removing container" podUID="759a1535-c79d-4518-ad7f-4219c9d28059" containerName="revoke-keys" Nov 03 15:30:07.354842 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.354794 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="759a1535-c79d-4518-ad7f-4219c9d28059" containerName="revoke-keys" Nov 03 15:30:07.354908 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.354899 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="759a1535-c79d-4518-ad7f-4219c9d28059" containerName="revoke-keys" Nov 03 15:30:07.358556 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.358540 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6" Nov 03 15:30:07.411991 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.411954 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:30:07.416380 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.416353 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6"] Nov 03 15:30:07.537396 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.537360 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a770e14-b6vj6\" (UID: \"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6" Nov 03 15:30:07.537396 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.537395 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a770e14-b6vj6\" (UID: \"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6" Nov 03 15:30:07.537686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.537626 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvkh5\" (UniqueName: \"kubernetes.io/projected/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-kube-api-access-zvkh5\") pod \"maas-api-revoke-sub-5a770e14-b6vj6\" (UID: \"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6" Nov 03 15:30:07.639001 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.638925 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvkh5\" (UniqueName: \"kubernetes.io/projected/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-kube-api-access-zvkh5\") pod \"maas-api-revoke-sub-5a770e14-b6vj6\" (UID: \"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6" Nov 03 15:30:07.639166 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.639044 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a770e14-b6vj6\" (UID: \"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6" Nov 03 15:30:07.639166 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.639061 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a770e14-b6vj6\" (UID: \"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6" Nov 03 15:30:07.639733 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.639707 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a770e14-b6vj6\" (UID: \"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6" Nov 03 15:30:07.641343 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.641322 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a770e14-b6vj6\" (UID: \"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6" Nov 03 15:30:07.703988 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.703956 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvkh5\" (UniqueName: \"kubernetes.io/projected/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-kube-api-access-zvkh5\") pod \"maas-api-revoke-sub-5a770e14-b6vj6\" (UID: \"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6" Nov 03 15:30:07.820192 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.820158 4557 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 15:30:07.820355 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.820210 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" podUID="82f7013a-125c-4b34-8f4b-b72f312179d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 15:30:07.972665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:07.972566 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6" Nov 03 15:30:08.094603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:08.093078 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:08.379610 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:30:08.379563 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a1d3e5_461b_46d5_bdaa_db9e79c802ef.slice/crio-5e002d93f0a879e96792e3abe7a0b7110d0ca0c6d4142f6f205f149008c02a55 WatchSource:0}: Error finding container 5e002d93f0a879e96792e3abe7a0b7110d0ca0c6d4142f6f205f149008c02a55: Status 404 returned error can't find the container with id 5e002d93f0a879e96792e3abe7a0b7110d0ca0c6d4142f6f205f149008c02a55 Nov 03 15:30:08.380043 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:08.379955 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6"] Nov 03 15:30:08.820393 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:08.820356 4557 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 15:30:08.820563 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:08.820420 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" podUID="82f7013a-125c-4b34-8f4b-b72f312179d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 15:30:09.131559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:09.131517 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6" event={"ID":"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef","Type":"ContainerStarted","Data":"1b71d64c5183e2f5f7a0bf2bd66cee8b34c40a0f7d19f00da4e32d5b53a5779a"} Nov 03 15:30:09.131802 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:09.131613 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6" event={"ID":"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef","Type":"ContainerStarted","Data":"5e002d93f0a879e96792e3abe7a0b7110d0ca0c6d4142f6f205f149008c02a55"} Nov 03 15:30:09.252067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:09.252013 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6" podStartSLOduration=2.251997412 podStartE2EDuration="2.251997412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:09.248769264 +0000 UTC m=+1319.091890808" watchObservedRunningTime="2026-11-03 15:30:09.251997412 +0000 UTC m=+1319.095118993" Nov 03 15:30:09.825036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:09.825007 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:09.825392 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:09.825349 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:09.825958 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:09.825938 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:30:10.140743 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:10.140649 4557 generic.go:356] "Generic (PLEG): container finished" podID="a9a1d3e5-461b-46d5-bdaa-db9e79c802ef" containerID="1b71d64c5183e2f5f7a0bf2bd66cee8b34c40a0f7d19f00da4e32d5b53a5779a" exitCode=0 Nov 03 15:30:10.140914 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:10.140752 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6" event={"ID":"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef","Type":"ContainerDied","Data":"1b71d64c5183e2f5f7a0bf2bd66cee8b34c40a0f7d19f00da4e32d5b53a5779a"} Nov 03 15:30:10.443164 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:10.443083 4557 scope.go:122] "RemoveContainer" containerID="16773dfce556eef987ec1d3b8e4d55c1ea15eeea1c3f195bb79c5af6e87293ea" Nov 03 15:30:10.452752 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:10.452735 4557 scope.go:122] "RemoveContainer" containerID="0b68ee73cb1aa17896dffc150ebf565fa45f5f6988c126f89f9f7130b5de3250" Nov 03 15:30:10.461725 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:10.461692 4557 scope.go:122] "RemoveContainer" containerID="5d7b363704716993df9ef96f05febb67c5657ec2dbec900a9e741184465353c7" Nov 03 15:30:10.473080 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:10.473059 4557 scope.go:122] "RemoveContainer" containerID="126999c00321b0fe194e3ce9f3f51cc133eea72cefd9cb73aa1cea9d9818f95e" Nov 03 15:30:10.483945 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:10.483923 4557 scope.go:122] "RemoveContainer" containerID="53d902445fd73a8acaacdfde00d625a0c44e20e35b7cfbcfec27bb1c6ce4f1a5" Nov 03 15:30:10.717728 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:10.717653 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:11.297022 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:11.296994 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6" Nov 03 15:30:11.386286 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:11.386254 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-maas-api-xks-ca\") pod \"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef\" (UID: \"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef\") " Nov 03 15:30:11.386462 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:11.386403 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-kube-api-access-zvkh5\" (UniqueName: \"kubernetes.io/projected/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-kube-api-access-zvkh5\") pod \"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef\" (UID: \"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef\") " Nov 03 15:30:11.386528 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:11.386510 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-maas-api-openshift-ca\") pod \"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef\" (UID: \"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef\") " Nov 03 15:30:11.386879 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:11.386848 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-maas-api-openshift-ca" pod "a9a1d3e5-461b-46d5-bdaa-db9e79c802ef" (UID: "a9a1d3e5-461b-46d5-bdaa-db9e79c802ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:11.387133 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:11.387113 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:11.388324 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:11.388300 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-maas-api-xks-ca" pod "a9a1d3e5-461b-46d5-bdaa-db9e79c802ef" (UID: "a9a1d3e5-461b-46d5-bdaa-db9e79c802ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:30:11.388441 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:11.388422 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-kube-api-access-zvkh5" pod "a9a1d3e5-461b-46d5-bdaa-db9e79c802ef" (UID: "a9a1d3e5-461b-46d5-bdaa-db9e79c802ef"). InnerVolumeSpecName "kube-api-access-zvkh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:11.488830 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:11.488749 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvkh5\" (UniqueName: \"kubernetes.io/projected/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-kube-api-access-zvkh5\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:11.488830 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:11.488778 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:12.156506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:12.156477 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6" Nov 03 15:30:12.156686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:12.156513 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6" event={"ID":"a9a1d3e5-461b-46d5-bdaa-db9e79c802ef","Type":"ContainerDied","Data":"5e002d93f0a879e96792e3abe7a0b7110d0ca0c6d4142f6f205f149008c02a55"} Nov 03 15:30:12.156686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:12.156595 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e002d93f0a879e96792e3abe7a0b7110d0ca0c6d4142f6f205f149008c02a55" Nov 03 15:30:13.050485 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:13.050450 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9"] Nov 03 15:30:13.100891 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:13.100859 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93ac7341-bczk9"] Nov 03 15:30:14.268722 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:14.268684 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:14.286344 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:14.286303 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ceb986-47b2-4326-ac8b-a664a425c81c" path="/var/lib/kubelet/pods/c5ceb986-47b2-4326-ac8b-a664a425c81c/volumes" Nov 03 15:30:15.279293 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.279261 4557 scope.go:122] "RemoveContainer" containerID="2a49f9870306216db8faf03288642bc88d9cb3ace38557394b84b7ef71ab22ab" Nov 03 15:30:15.318545 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.318515 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:15.376620 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.374181 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl"] Nov 03 15:30:15.376620 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.375242 4557 cpu_manager.go:414] "containerMap: removing container" podUID="a9a1d3e5-461b-46d5-bdaa-db9e79c802ef" containerName="revoke-keys" Nov 03 15:30:15.376620 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.375267 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a1d3e5-461b-46d5-bdaa-db9e79c802ef" containerName="revoke-keys" Nov 03 15:30:15.376620 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.375492 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a1d3e5-461b-46d5-bdaa-db9e79c802ef" containerName="revoke-keys" Nov 03 15:30:15.381797 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.381770 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl" Nov 03 15:30:15.430620 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.430554 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl"] Nov 03 15:30:15.435695 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.435668 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:30:15.451350 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.451317 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8f0988-165b-4bee-85c2-357bfb4e4a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b85f9f9-zjhbl\" (UID: \"5a8f0988-165b-4bee-85c2-357bfb4e4a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl" Nov 03 15:30:15.451515 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.451491 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8f0988-165b-4bee-85c2-357bfb4e4a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b85f9f9-zjhbl\" (UID: \"5a8f0988-165b-4bee-85c2-357bfb4e4a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl" Nov 03 15:30:15.451708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.451690 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrxjv\" (UniqueName: \"kubernetes.io/projected/5a8f0988-165b-4bee-85c2-357bfb4e4a60-kube-api-access-mrxjv\") pod \"maas-api-revoke-sub-5b85f9f9-zjhbl\" (UID: \"5a8f0988-165b-4bee-85c2-357bfb4e4a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl" Nov 03 15:30:15.552627 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.552575 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrxjv\" (UniqueName: \"kubernetes.io/projected/5a8f0988-165b-4bee-85c2-357bfb4e4a60-kube-api-access-mrxjv\") pod \"maas-api-revoke-sub-5b85f9f9-zjhbl\" (UID: \"5a8f0988-165b-4bee-85c2-357bfb4e4a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl" Nov 03 15:30:15.552802 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.552696 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8f0988-165b-4bee-85c2-357bfb4e4a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b85f9f9-zjhbl\" (UID: \"5a8f0988-165b-4bee-85c2-357bfb4e4a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl" Nov 03 15:30:15.552861 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.552815 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8f0988-165b-4bee-85c2-357bfb4e4a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b85f9f9-zjhbl\" (UID: \"5a8f0988-165b-4bee-85c2-357bfb4e4a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl" Nov 03 15:30:15.553427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.553399 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8f0988-165b-4bee-85c2-357bfb4e4a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b85f9f9-zjhbl\" (UID: \"5a8f0988-165b-4bee-85c2-357bfb4e4a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl" Nov 03 15:30:15.559245 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.559213 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8f0988-165b-4bee-85c2-357bfb4e4a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b85f9f9-zjhbl\" (UID: \"5a8f0988-165b-4bee-85c2-357bfb4e4a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl" Nov 03 15:30:15.616281 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.616252 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrxjv\" (UniqueName: \"kubernetes.io/projected/5a8f0988-165b-4bee-85c2-357bfb4e4a60-kube-api-access-mrxjv\") pod \"maas-api-revoke-sub-5b85f9f9-zjhbl\" (UID: \"5a8f0988-165b-4bee-85c2-357bfb4e4a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl" Nov 03 15:30:15.638629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.638594 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:15.697902 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.697877 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl" Nov 03 15:30:15.834365 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:15.834258 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:16.068202 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:16.068163 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8"] Nov 03 15:30:16.076367 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:16.076337 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b378ea-pllv8"] Nov 03 15:30:16.107331 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:30:16.107297 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8f0988_165b_4bee_85c2_357bfb4e4a60.slice/crio-b6350bdb02cb2d7cca10def2c1d646215fbd51f657b25287ab14996ac6e1cbce WatchSource:0}: Error finding container b6350bdb02cb2d7cca10def2c1d646215fbd51f657b25287ab14996ac6e1cbce: Status 404 returned error can't find the container with id b6350bdb02cb2d7cca10def2c1d646215fbd51f657b25287ab14996ac6e1cbce Nov 03 15:30:16.107820 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:16.107798 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl"] Nov 03 15:30:16.197018 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:16.196984 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" event={"ID":"be6660d4-9d6b-4706-9cfd-0d204e1b4e45","Type":"ContainerStarted","Data":"1932430bf77eef129e1aeebe9be58a94423ecb9af6a1a78236ed465a2aac61f3"} Nov 03 15:30:16.198760 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:16.198725 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl" event={"ID":"5a8f0988-165b-4bee-85c2-357bfb4e4a60","Type":"ContainerStarted","Data":"b6350bdb02cb2d7cca10def2c1d646215fbd51f657b25287ab14996ac6e1cbce"} Nov 03 15:30:16.278976 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:16.278938 4557 scope.go:122] "RemoveContainer" containerID="b7256769e53da54b80c62f72af2adf06e70690a32a26013e9dda1313e8b37e73" Nov 03 15:30:16.279155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:16.279133 4557 scope.go:122] "RemoveContainer" containerID="e82c32cecdcc83b9c4d3d655403fe8116fb0305bcccd9e24ab01d30fb2febf07" Nov 03 15:30:16.289083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:16.289056 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="382b7b36-beb8-4b8d-aa48-4071c3cd4f10" path="/var/lib/kubelet/pods/382b7b36-beb8-4b8d-aa48-4071c3cd4f10/volumes" Nov 03 15:30:16.531062 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:30:16.531027 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:30:16.650980 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:16.650895 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:17.210071 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.210037 4557 generic.go:356] "Generic (PLEG): container finished" podID="5a8f0988-165b-4bee-85c2-357bfb4e4a60" containerID="6cd2d6c4a92e6ff1dd9b55f6f6a33c4e46bd22a1694f6b00caabf54f79a6e046" exitCode=0 Nov 03 15:30:17.210204 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.210131 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl" event={"ID":"5a8f0988-165b-4bee-85c2-357bfb4e4a60","Type":"ContainerDied","Data":"6cd2d6c4a92e6ff1dd9b55f6f6a33c4e46bd22a1694f6b00caabf54f79a6e046"} Nov 03 15:30:17.212438 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.212414 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" event={"ID":"192929b4-7ed3-4e9d-9a32-f18088dc5910","Type":"ContainerStarted","Data":"d81b07cbcb8a8c7ac3e00994cc171cd28762a95785d187bb31d4801728d8bdd6"} Nov 03 15:30:17.214468 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.214447 4557 generic.go:356] "Generic (PLEG): container finished" podID="c7b8f36c-0d65-4523-b767-cd6970eb594e" containerID="b89e3fedb837762dead6b20252a43169dbd1cb3e4cc0226e6308ba50c5b3cf03" exitCode=60 Nov 03 15:30:17.214569 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.214524 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" event={"ID":"c7b8f36c-0d65-4523-b767-cd6970eb594e","Type":"ContainerDied","Data":"b89e3fedb837762dead6b20252a43169dbd1cb3e4cc0226e6308ba50c5b3cf03"} Nov 03 15:30:17.214714 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.214573 4557 scope.go:122] "RemoveContainer" containerID="b7256769e53da54b80c62f72af2adf06e70690a32a26013e9dda1313e8b37e73" Nov 03 15:30:17.214937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.214917 4557 scope.go:122] "RemoveContainer" containerID="b89e3fedb837762dead6b20252a43169dbd1cb3e4cc0226e6308ba50c5b3cf03" Nov 03 15:30:17.215232 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:30:17.215206 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9_odh-ai-gateway-infra(c7b8f36c-0d65-4523-b767-cd6970eb594e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" podUID="c7b8f36c-0d65-4523-b767-cd6970eb594e" Nov 03 15:30:17.216680 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.216662 4557 generic.go:356] "Generic (PLEG): container finished" podID="be6660d4-9d6b-4706-9cfd-0d204e1b4e45" containerID="1932430bf77eef129e1aeebe9be58a94423ecb9af6a1a78236ed465a2aac61f3" exitCode=60 Nov 03 15:30:17.216765 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.216731 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" event={"ID":"be6660d4-9d6b-4706-9cfd-0d204e1b4e45","Type":"ContainerDied","Data":"1932430bf77eef129e1aeebe9be58a94423ecb9af6a1a78236ed465a2aac61f3"} Nov 03 15:30:17.217010 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.216997 4557 scope.go:122] "RemoveContainer" containerID="1932430bf77eef129e1aeebe9be58a94423ecb9af6a1a78236ed465a2aac61f3" Nov 03 15:30:17.217221 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:30:17.217205 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx_odh-ai-gateway-infra(be6660d4-9d6b-4706-9cfd-0d204e1b4e45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" podUID="be6660d4-9d6b-4706-9cfd-0d204e1b4e45" Nov 03 15:30:17.226825 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.226808 4557 scope.go:122] "RemoveContainer" containerID="2a49f9870306216db8faf03288642bc88d9cb3ace38557394b84b7ef71ab22ab" Nov 03 15:30:17.264643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.264605 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:17.319315 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.319261 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" podStartSLOduration=17.319247622 podStartE2EDuration="17.319247622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:16.321387808 +0000 UTC m=+1326.164509349" watchObservedRunningTime="2026-11-03 15:30:17.319247622 +0000 UTC m=+1327.162369206" Nov 03 15:30:17.323451 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.323429 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z"] Nov 03 15:30:17.327434 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.327383 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.373456 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.373431 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx"] Nov 03 15:30:17.377361 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.377340 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb"] Nov 03 15:30:17.381573 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.381551 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.394068 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.394033 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv7p2\" (UniqueName: \"kubernetes.io/projected/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-kube-api-access-tv7p2\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.394245 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.394098 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.394245 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.394125 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.394245 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.394148 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.394497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.394253 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.394497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.394281 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.394497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.394330 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.394497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.394387 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.394497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.394427 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.399850 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.399814 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z"] Nov 03 15:30:17.470792 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.470760 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb"] Nov 03 15:30:17.495597 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.495555 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.495742 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.495609 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.495742 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.495633 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.495742 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.495661 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.495742 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.495705 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.495907 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.495801 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.495907 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.495839 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.495907 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.495889 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qczms\" (UniqueName: \"kubernetes.io/projected/e362b475-7544-4095-a465-24905a71c6c0-kube-api-access-qczms\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.496043 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.495922 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.496043 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.496007 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.496155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.496054 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.496259 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.496236 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e362b475-7544-4095-a465-24905a71c6c0-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.496327 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.496284 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.497005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.496509 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.497005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.496633 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.497005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.496684 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.497005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.496695 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.497005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.496741 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.497005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.496817 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e362b475-7544-4095-a465-24905a71c6c0-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.497005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.496936 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e362b475-7544-4095-a465-24905a71c6c0-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.497005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.496957 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.497514 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.497022 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv7p2\" (UniqueName: \"kubernetes.io/projected/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-kube-api-access-tv7p2\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.497514 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.497335 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.498166 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.498144 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.498784 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.498761 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.558566 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.558522 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.558764 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.558656 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv7p2\" (UniqueName: \"kubernetes.io/projected/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-kube-api-access-tv7p2\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.598919 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.598815 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.598919 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.598882 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.599132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.599007 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qczms\" (UniqueName: \"kubernetes.io/projected/e362b475-7544-4095-a465-24905a71c6c0-kube-api-access-qczms\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.599132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.599037 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.599132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.599065 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e362b475-7544-4095-a465-24905a71c6c0-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.599132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.599092 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.599132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.599121 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.599314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.599210 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e362b475-7544-4095-a465-24905a71c6c0-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.599314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.599249 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.599314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.599288 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e362b475-7544-4095-a465-24905a71c6c0-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.599926 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.599764 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.599926 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.599882 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.599926 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.599897 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e362b475-7544-4095-a465-24905a71c6c0-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.600159 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.600021 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.601742 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.601716 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.602015 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.601993 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e362b475-7544-4095-a465-24905a71c6c0-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.615433 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.615408 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc"] Nov 03 15:30:17.621166 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.621148 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.663942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.663909 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e362b475-7544-4095-a465-24905a71c6c0-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.666572 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.665253 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:17.668604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.668518 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qczms\" (UniqueName: \"kubernetes.io/projected/e362b475-7544-4095-a465-24905a71c6c0-kube-api-access-qczms\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.684368 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.683788 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc"] Nov 03 15:30:17.706643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.702343 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f82529-9942-46d4-98dc-fa872122b973-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.706643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.702408 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.706643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.702432 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.706643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.702490 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.706643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.702604 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.706643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.702644 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.706643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.702720 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxx46\" (UniqueName: \"kubernetes.io/projected/58f82529-9942-46d4-98dc-fa872122b973-kube-api-access-xxx46\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.706643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.702750 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f82529-9942-46d4-98dc-fa872122b973-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.706643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.702825 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f82529-9942-46d4-98dc-fa872122b973-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.706643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.703058 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:17.804480 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.804442 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f82529-9942-46d4-98dc-fa872122b973-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.804689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.804521 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f82529-9942-46d4-98dc-fa872122b973-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.804689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.804552 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.804689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.804570 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.804689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.804639 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.804909 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.804719 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.804909 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.804745 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.804909 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.804800 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxx46\" (UniqueName: \"kubernetes.io/projected/58f82529-9942-46d4-98dc-fa872122b973-kube-api-access-xxx46\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.804909 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.804819 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f82529-9942-46d4-98dc-fa872122b973-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.805820 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.805444 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f82529-9942-46d4-98dc-fa872122b973-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.805820 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.805425 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.805820 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.805767 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.805820 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.805796 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.806100 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.805930 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.807660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.807640 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.807770 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.807711 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f82529-9942-46d4-98dc-fa872122b973-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.891416 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.886630 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f82529-9942-46d4-98dc-fa872122b973-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.891416 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.890443 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxx46\" (UniqueName: \"kubernetes.io/projected/58f82529-9942-46d4-98dc-fa872122b973-kube-api-access-xxx46\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:17.944192 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:17.944154 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:18.026712 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.026679 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9"] Nov 03 15:30:18.033356 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.033107 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.097656 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.097620 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9"] Nov 03 15:30:18.114965 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.114922 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.115239 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.114978 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.115239 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.115026 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.115239 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.115050 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.115239 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.115128 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.115239 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.115193 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.115695 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.115249 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvqqh\" (UniqueName: \"kubernetes.io/projected/d6d2a233-488a-42d3-8e7b-cd7868a6f167-kube-api-access-qvqqh\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.115695 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.115339 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.115695 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.115383 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.201747 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:30:18.201192 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8227fd_da10_4bd8_b86f_5f7fc48ef40c.slice/crio-7a1fb69e11b1a9b059c8e1f2e1d03d1287f28d2c2e34e7680112bf4b1dfa6754 WatchSource:0}: Error finding container 7a1fb69e11b1a9b059c8e1f2e1d03d1287f28d2c2e34e7680112bf4b1dfa6754: Status 404 returned error can't find the container with id 7a1fb69e11b1a9b059c8e1f2e1d03d1287f28d2c2e34e7680112bf4b1dfa6754 Nov 03 15:30:18.201982 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.201925 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb"] Nov 03 15:30:18.203054 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:30:18.203029 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode362b475_7544_4095_a465_24905a71c6c0.slice/crio-10f5465e8fa091ae3f6a09414e08e3d4b1541b11b2ee30fd844f34c669553ddb WatchSource:0}: Error finding container 10f5465e8fa091ae3f6a09414e08e3d4b1541b11b2ee30fd844f34c669553ddb: Status 404 returned error can't find the container with id 10f5465e8fa091ae3f6a09414e08e3d4b1541b11b2ee30fd844f34c669553ddb Nov 03 15:30:18.205770 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.205676 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:18.205770 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.205751 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:18.205962 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.205790 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:18.212688 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.210853 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:18.212688 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.211448 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:18.212688 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.211858 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:18.228807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.225943 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.228807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.226043 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.228807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.226096 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.228807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.226154 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.228807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.226232 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.228807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.226298 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.228807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.226354 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvqqh\" (UniqueName: \"kubernetes.io/projected/d6d2a233-488a-42d3-8e7b-cd7868a6f167-kube-api-access-qvqqh\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.228807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.226513 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.228807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.226555 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.230464 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.230406 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.231017 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.230996 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.234138 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.231330 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.234138 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.231965 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.234138 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.233694 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z"] Nov 03 15:30:18.234138 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.233746 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.241318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.241299 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.241437 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.241418 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.259082 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.259038 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" event={"ID":"e362b475-7544-4095-a465-24905a71c6c0","Type":"ContainerStarted","Data":"10f5465e8fa091ae3f6a09414e08e3d4b1541b11b2ee30fd844f34c669553ddb"} Nov 03 15:30:18.261886 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.261855 4557 generic.go:356] "Generic (PLEG): container finished" podID="192929b4-7ed3-4e9d-9a32-f18088dc5910" containerID="d81b07cbcb8a8c7ac3e00994cc171cd28762a95785d187bb31d4801728d8bdd6" exitCode=60 Nov 03 15:30:18.261989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.261967 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" event={"ID":"192929b4-7ed3-4e9d-9a32-f18088dc5910","Type":"ContainerDied","Data":"d81b07cbcb8a8c7ac3e00994cc171cd28762a95785d187bb31d4801728d8bdd6"} Nov 03 15:30:18.262044 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.262035 4557 scope.go:122] "RemoveContainer" containerID="e82c32cecdcc83b9c4d3d655403fe8116fb0305bcccd9e24ab01d30fb2febf07" Nov 03 15:30:18.262593 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.262554 4557 scope.go:122] "RemoveContainer" containerID="d81b07cbcb8a8c7ac3e00994cc171cd28762a95785d187bb31d4801728d8bdd6" Nov 03 15:30:18.263049 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:30:18.262968 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg_odh-ai-gateway-infra(192929b4-7ed3-4e9d-9a32-f18088dc5910)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" podUID="192929b4-7ed3-4e9d-9a32-f18088dc5910" Nov 03 15:30:18.266765 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.266733 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" event={"ID":"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c","Type":"ContainerStarted","Data":"7a1fb69e11b1a9b059c8e1f2e1d03d1287f28d2c2e34e7680112bf4b1dfa6754"} Nov 03 15:30:18.323246 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.323188 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.337520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.337446 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvqqh\" (UniqueName: \"kubernetes.io/projected/d6d2a233-488a-42d3-8e7b-cd7868a6f167-kube-api-access-qvqqh\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.448681 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.408882 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:18.449392 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:30:18.449343 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f82529_9942_46d4_98dc_fa872122b973.slice/crio-5f2d8ca1177eeb230db785587c3be524829fd925947403d5ff2684e9a71ac00c WatchSource:0}: Error finding container 5f2d8ca1177eeb230db785587c3be524829fd925947403d5ff2684e9a71ac00c: Status 404 returned error can't find the container with id 5f2d8ca1177eeb230db785587c3be524829fd925947403d5ff2684e9a71ac00c Nov 03 15:30:18.462079 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.462046 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc"] Nov 03 15:30:18.572345 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.572300 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:18.576648 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.572640 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:18.577052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.577019 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:18.607379 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.607354 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" Nov 03 15:30:18.659679 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.651827 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6660d4-9d6b-4706-9cfd-0d204e1b4e45-kube-api-access-jpnhm\" (UniqueName: \"kubernetes.io/projected/be6660d4-9d6b-4706-9cfd-0d204e1b4e45-kube-api-access-jpnhm\") pod \"be6660d4-9d6b-4706-9cfd-0d204e1b4e45\" (UID: \"be6660d4-9d6b-4706-9cfd-0d204e1b4e45\") " Nov 03 15:30:18.659679 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.651897 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be6660d4-9d6b-4706-9cfd-0d204e1b4e45-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6660d4-9d6b-4706-9cfd-0d204e1b4e45-maas-api-service-ca\") pod \"be6660d4-9d6b-4706-9cfd-0d204e1b4e45\" (UID: \"be6660d4-9d6b-4706-9cfd-0d204e1b4e45\") " Nov 03 15:30:18.659679 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.653042 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6660d4-9d6b-4706-9cfd-0d204e1b4e45-maas-api-service-ca" pod "be6660d4-9d6b-4706-9cfd-0d204e1b4e45" (UID: "be6660d4-9d6b-4706-9cfd-0d204e1b4e45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:18.660145 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.659827 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6660d4-9d6b-4706-9cfd-0d204e1b4e45-kube-api-access-jpnhm" pod "be6660d4-9d6b-4706-9cfd-0d204e1b4e45" (UID: "be6660d4-9d6b-4706-9cfd-0d204e1b4e45"). InnerVolumeSpecName "kube-api-access-jpnhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:18.667069 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.667024 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8"] Nov 03 15:30:18.674490 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.674465 4557 cpu_manager.go:414] "containerMap: removing container" podUID="be6660d4-9d6b-4706-9cfd-0d204e1b4e45" containerName="cleanup" Nov 03 15:30:18.674684 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.674670 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6660d4-9d6b-4706-9cfd-0d204e1b4e45" containerName="cleanup" Nov 03 15:30:18.674779 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.674769 4557 cpu_manager.go:414] "containerMap: removing container" podUID="be6660d4-9d6b-4706-9cfd-0d204e1b4e45" containerName="cleanup" Nov 03 15:30:18.674847 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.674838 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6660d4-9d6b-4706-9cfd-0d204e1b4e45" containerName="cleanup" Nov 03 15:30:18.674929 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.674918 4557 cpu_manager.go:414] "containerMap: removing container" podUID="be6660d4-9d6b-4706-9cfd-0d204e1b4e45" containerName="cleanup" Nov 03 15:30:18.675008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.674998 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6660d4-9d6b-4706-9cfd-0d204e1b4e45" containerName="cleanup" Nov 03 15:30:18.679436 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.679420 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6660d4-9d6b-4706-9cfd-0d204e1b4e45" containerName="cleanup" Nov 03 15:30:18.679544 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.679535 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6660d4-9d6b-4706-9cfd-0d204e1b4e45" containerName="cleanup" Nov 03 15:30:18.679649 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.679639 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6660d4-9d6b-4706-9cfd-0d204e1b4e45" containerName="cleanup" Nov 03 15:30:18.689293 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.689271 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.709688 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.709665 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9"] Nov 03 15:30:18.779128 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.762951 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-d785c53b-openshift-default-dockercfg-mnjnh\"" Nov 03 15:30:18.779128 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.765550 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpnhm\" (UniqueName: \"kubernetes.io/projected/be6660d4-9d6b-4706-9cfd-0d204e1b4e45-kube-api-access-jpnhm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:18.810942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.791613 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6660d4-9d6b-4706-9cfd-0d204e1b4e45-maas-api-service-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:18.865772 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.865721 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:18.869713 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.869656 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8"] Nov 03 15:30:18.895232 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.892907 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-token\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.895232 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.892992 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-credential-socket\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.895232 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.893125 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-workload-certs\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.895232 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.893154 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-podinfo\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.902609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.895629 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrd7n\" (UniqueName: \"kubernetes.io/projected/4b57651a-2c8b-4df4-84c2-4f5fba809417-kube-api-access-hrd7n\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.902609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.895694 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-data\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.902609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.896064 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-workload-socket\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.902609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.896097 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-envoy\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.902609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.896150 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b57651a-2c8b-4df4-84c2-4f5fba809417-istiod-ca-cert\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.976514 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.976369 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl" Nov 03 15:30:18.983892 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.981859 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg"] Nov 03 15:30:18.999754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.997704 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a8f0988-165b-4bee-85c2-357bfb4e4a60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8f0988-165b-4bee-85c2-357bfb4e4a60-maas-api-openshift-ca\") pod \"5a8f0988-165b-4bee-85c2-357bfb4e4a60\" (UID: \"5a8f0988-165b-4bee-85c2-357bfb4e4a60\") " Nov 03 15:30:18.999754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.997814 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a8f0988-165b-4bee-85c2-357bfb4e4a60-kube-api-access-mrxjv\" (UniqueName: \"kubernetes.io/projected/5a8f0988-165b-4bee-85c2-357bfb4e4a60-kube-api-access-mrxjv\") pod \"5a8f0988-165b-4bee-85c2-357bfb4e4a60\" (UID: \"5a8f0988-165b-4bee-85c2-357bfb4e4a60\") " Nov 03 15:30:18.999754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.998025 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a8f0988-165b-4bee-85c2-357bfb4e4a60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8f0988-165b-4bee-85c2-357bfb4e4a60-maas-api-xks-ca\") pod \"5a8f0988-165b-4bee-85c2-357bfb4e4a60\" (UID: \"5a8f0988-165b-4bee-85c2-357bfb4e4a60\") " Nov 03 15:30:18.999754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.998264 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-token\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.999754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.998332 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-credential-socket\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.999754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.998429 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-workload-certs\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.999754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.998454 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-podinfo\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.999754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.998480 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrd7n\" (UniqueName: \"kubernetes.io/projected/4b57651a-2c8b-4df4-84c2-4f5fba809417-kube-api-access-hrd7n\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.999754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.998519 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-data\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.999754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.998552 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-workload-socket\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.999754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.998596 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-envoy\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.999754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.998647 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b57651a-2c8b-4df4-84c2-4f5fba809417-istiod-ca-cert\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:18.999754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:18.999544 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b57651a-2c8b-4df4-84c2-4f5fba809417-istiod-ca-cert\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:19.000995 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.000708 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a8f0988-165b-4bee-85c2-357bfb4e4a60-maas-api-openshift-ca" pod "5a8f0988-165b-4bee-85c2-357bfb4e4a60" (UID: "5a8f0988-165b-4bee-85c2-357bfb4e4a60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:19.007960 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.005437 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-credential-socket\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:19.007960 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.005709 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-workload-certs\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:19.007960 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.005932 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-data\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:19.007960 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.006395 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-workload-socket\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:19.014617 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.009259 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-envoy\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:19.046868 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.046789 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8f0988-165b-4bee-85c2-357bfb4e4a60-kube-api-access-mrxjv" pod "5a8f0988-165b-4bee-85c2-357bfb4e4a60" (UID: "5a8f0988-165b-4bee-85c2-357bfb4e4a60"). InnerVolumeSpecName "kube-api-access-mrxjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:19.049627 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.048749 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a8f0988-165b-4bee-85c2-357bfb4e4a60-maas-api-xks-ca" pod "5a8f0988-165b-4bee-85c2-357bfb4e4a60" (UID: "5a8f0988-165b-4bee-85c2-357bfb4e4a60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:30:19.058915 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.058887 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-podinfo\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:19.092143 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.092006 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-token\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:19.100813 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.100780 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8f0988-165b-4bee-85c2-357bfb4e4a60-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:19.101019 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.101001 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8f0988-165b-4bee-85c2-357bfb4e4a60-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:19.101156 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.101142 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrxjv\" (UniqueName: \"kubernetes.io/projected/5a8f0988-165b-4bee-85c2-357bfb4e4a60-kube-api-access-mrxjv\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:19.110688 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.110622 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrd7n\" (UniqueName: \"kubernetes.io/projected/4b57651a-2c8b-4df4-84c2-4f5fba809417-kube-api-access-hrd7n\") pod \"e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:19.127100 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.127061 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:19.238774 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.238366 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" Nov 03 15:30:19.240430 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.240402 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9"] Nov 03 15:30:19.241932 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:30:19.241904 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d2a233_488a_42d3_8e7b_cd7868a6f167.slice/crio-cbd53a3dfbda81be1073f1ef949b73d1b407ac168ad9c6802fc804a4864d2a7c WatchSource:0}: Error finding container cbd53a3dfbda81be1073f1ef949b73d1b407ac168ad9c6802fc804a4864d2a7c: Status 404 returned error can't find the container with id cbd53a3dfbda81be1073f1ef949b73d1b407ac168ad9c6802fc804a4864d2a7c Nov 03 15:30:19.251764 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.251739 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:19.314343 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.314303 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b8f36c-0d65-4523-b767-cd6970eb594e-kube-api-access-9w5n9\" (UniqueName: \"kubernetes.io/projected/c7b8f36c-0d65-4523-b767-cd6970eb594e-kube-api-access-9w5n9\") pod \"c7b8f36c-0d65-4523-b767-cd6970eb594e\" (UID: \"c7b8f36c-0d65-4523-b767-cd6970eb594e\") " Nov 03 15:30:19.314541 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.314475 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b8f36c-0d65-4523-b767-cd6970eb594e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b8f36c-0d65-4523-b767-cd6970eb594e-maas-api-service-ca\") pod \"c7b8f36c-0d65-4523-b767-cd6970eb594e\" (UID: \"c7b8f36c-0d65-4523-b767-cd6970eb594e\") " Nov 03 15:30:19.325624 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.325573 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b8f36c-0d65-4523-b767-cd6970eb594e-maas-api-service-ca" pod "c7b8f36c-0d65-4523-b767-cd6970eb594e" (UID: "c7b8f36c-0d65-4523-b767-cd6970eb594e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:19.345939 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.345849 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b8f36c-0d65-4523-b767-cd6970eb594e-kube-api-access-9w5n9" pod "c7b8f36c-0d65-4523-b767-cd6970eb594e" (UID: "c7b8f36c-0d65-4523-b767-cd6970eb594e"). InnerVolumeSpecName "kube-api-access-9w5n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:19.386646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.368131 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:19.386646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.368224 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:19.386646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.368270 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:19.387954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.387789 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:19.423130 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.423043 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w5n9\" (UniqueName: \"kubernetes.io/projected/c7b8f36c-0d65-4523-b767-cd6970eb594e-kube-api-access-9w5n9\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:19.423130 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.423079 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b8f36c-0d65-4523-b767-cd6970eb594e-maas-api-service-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:19.672224 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.672187 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl" event={"ID":"5a8f0988-165b-4bee-85c2-357bfb4e4a60","Type":"ContainerDied","Data":"b6350bdb02cb2d7cca10def2c1d646215fbd51f657b25287ab14996ac6e1cbce"} Nov 03 15:30:19.674639 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.674610 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6350bdb02cb2d7cca10def2c1d646215fbd51f657b25287ab14996ac6e1cbce" Nov 03 15:30:19.675607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.675571 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl" Nov 03 15:30:19.748082 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.748034 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" event={"ID":"d6d2a233-488a-42d3-8e7b-cd7868a6f167","Type":"ContainerStarted","Data":"cbd53a3dfbda81be1073f1ef949b73d1b407ac168ad9c6802fc804a4864d2a7c"} Nov 03 15:30:19.766332 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.766245 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" event={"ID":"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c","Type":"ContainerStarted","Data":"dc73687addedf98708f783dfaafb06080d641677899ec127943fc2b696cb2d88"} Nov 03 15:30:19.776021 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.775056 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" event={"ID":"58f82529-9942-46d4-98dc-fa872122b973","Type":"ContainerStarted","Data":"967125a7ab4054954fe4017938510f3ad637912912db8ec2573dd3b19cb475ba"} Nov 03 15:30:19.776021 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.775111 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" event={"ID":"58f82529-9942-46d4-98dc-fa872122b973","Type":"ContainerStarted","Data":"5f2d8ca1177eeb230db785587c3be524829fd925947403d5ff2684e9a71ac00c"} Nov 03 15:30:19.833976 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.819027 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" event={"ID":"c7b8f36c-0d65-4523-b767-cd6970eb594e","Type":"ContainerDied","Data":"cd991d968e7da45065906e6006b5fa3f421381722a560d1d87fb11474559503f"} Nov 03 15:30:19.833976 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.819099 4557 scope.go:122] "RemoveContainer" containerID="b89e3fedb837762dead6b20252a43169dbd1cb3e4cc0226e6308ba50c5b3cf03" Nov 03 15:30:19.833976 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.819238 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9" Nov 03 15:30:19.953743 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.953714 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:19.958739 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.958708 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:19.958949 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.958923 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" podUID="58f82529-9942-46d4-98dc-fa872122b973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 15:30:20.017247 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.998549 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" event={"ID":"be6660d4-9d6b-4706-9cfd-0d204e1b4e45","Type":"ContainerDied","Data":"43ce364767919e715915e7978463fd00ed51dadc70daf10dbdf8c0b26fec6a75"} Nov 03 15:30:20.017247 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:19.998703 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx" Nov 03 15:30:20.053289 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.053253 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" event={"ID":"e362b475-7544-4095-a465-24905a71c6c0","Type":"ContainerStarted","Data":"9387861b5b3928ba16c1e26e80a13ef6941619ab5f09fcf231a13bb8bbc39874"} Nov 03 15:30:20.068670 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.068643 4557 scope.go:122] "RemoveContainer" containerID="1932430bf77eef129e1aeebe9be58a94423ecb9af6a1a78236ed465a2aac61f3" Nov 03 15:30:20.152990 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.152962 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8"] Nov 03 15:30:20.155402 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:30:20.155377 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b57651a_2c8b_4df4_84c2_4f5fba809417.slice/crio-6400399bb8446061e1e35c6262cf1fac35d078822a9137dafa221a9207b30923 WatchSource:0}: Error finding container 6400399bb8446061e1e35c6262cf1fac35d078822a9137dafa221a9207b30923: Status 404 returned error can't find the container with id 6400399bb8446061e1e35c6262cf1fac35d078822a9137dafa221a9207b30923 Nov 03 15:30:20.158813 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.158785 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:20.158891 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.158848 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:20.158891 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.158883 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:20.202656 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.176874 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" podStartSLOduration=3.176856686 podStartE2EDuration="3.176856686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:20.175268284 +0000 UTC m=+1330.018389833" watchObservedRunningTime="2026-11-03 15:30:20.176856686 +0000 UTC m=+1330.019978234" Nov 03 15:30:20.356374 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.356301 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" podStartSLOduration=3.356280543 podStartE2EDuration="3.356280543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:20.325661362 +0000 UTC m=+1330.168782900" watchObservedRunningTime="2026-11-03 15:30:20.356280543 +0000 UTC m=+1330.199402086" Nov 03 15:30:20.370045 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.370019 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" Nov 03 15:30:20.443864 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.443821 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/192929b4-7ed3-4e9d-9a32-f18088dc5910-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/192929b4-7ed3-4e9d-9a32-f18088dc5910-maas-api-service-ca\") pod \"192929b4-7ed3-4e9d-9a32-f18088dc5910\" (UID: \"192929b4-7ed3-4e9d-9a32-f18088dc5910\") " Nov 03 15:30:20.443987 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.443884 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192929b4-7ed3-4e9d-9a32-f18088dc5910-kube-api-access-whw7h\" (UniqueName: \"kubernetes.io/projected/192929b4-7ed3-4e9d-9a32-f18088dc5910-kube-api-access-whw7h\") pod \"192929b4-7ed3-4e9d-9a32-f18088dc5910\" (UID: \"192929b4-7ed3-4e9d-9a32-f18088dc5910\") " Nov 03 15:30:20.444284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.444257 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192929b4-7ed3-4e9d-9a32-f18088dc5910-maas-api-service-ca" pod "192929b4-7ed3-4e9d-9a32-f18088dc5910" (UID: "192929b4-7ed3-4e9d-9a32-f18088dc5910"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:20.444626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.444604 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/192929b4-7ed3-4e9d-9a32-f18088dc5910-maas-api-service-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:20.446081 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.446056 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192929b4-7ed3-4e9d-9a32-f18088dc5910-kube-api-access-whw7h" pod "192929b4-7ed3-4e9d-9a32-f18088dc5910" (UID: "192929b4-7ed3-4e9d-9a32-f18088dc5910"). InnerVolumeSpecName "kube-api-access-whw7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:20.552623 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.524042 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9"] Nov 03 15:30:20.569508 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.563653 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whw7h\" (UniqueName: \"kubernetes.io/projected/192929b4-7ed3-4e9d-9a32-f18088dc5910-kube-api-access-whw7h\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:20.619150 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.619066 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895330-ktlx9"] Nov 03 15:30:20.695484 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.684060 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:20.695866 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.695814 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 15:30:20.696052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.696025 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" podUID="3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 15:30:20.707983 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.707952 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:20.708218 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.708144 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:20.708346 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.708325 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" podUID="e362b475-7544-4095-a465-24905a71c6c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 15:30:20.746359 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.746292 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" podStartSLOduration=3.746271705 podStartE2EDuration="3.746271705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:20.730008931 +0000 UTC m=+1330.573130474" watchObservedRunningTime="2026-11-03 15:30:20.746271705 +0000 UTC m=+1330.589393244" Nov 03 15:30:20.953337 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.953166 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:20.953337 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:20.953222 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" podUID="58f82529-9942-46d4-98dc-fa872122b973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 15:30:21.071870 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:21.071842 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx"] Nov 03 15:30:21.338605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:21.334742 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895330-9zzkx"] Nov 03 15:30:21.374702 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:21.374651 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" event={"ID":"192929b4-7ed3-4e9d-9a32-f18088dc5910","Type":"ContainerDied","Data":"e338e868a90e9ded1796bd8bd214c8dc60dc35eb77f055e05b3cddbabf8769a8"} Nov 03 15:30:21.414914 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:21.414881 4557 scope.go:122] "RemoveContainer" containerID="d81b07cbcb8a8c7ac3e00994cc171cd28762a95785d187bb31d4801728d8bdd6" Nov 03 15:30:21.415356 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:21.374918 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg" Nov 03 15:30:21.573023 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:21.564632 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:21.657661 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:21.648702 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:21.707610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:21.707298 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 15:30:21.707610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:21.707375 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" podUID="3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 15:30:21.730340 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:21.730298 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" event={"ID":"4b57651a-2c8b-4df4-84c2-4f5fba809417","Type":"ContainerStarted","Data":"e6c9817163d5e9bd3b616de0cc4ce8359718ab7abd15f2e98c58b02312bc73cb"} Nov 03 15:30:21.753823 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:21.743818 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:21.753823 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:21.752762 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" podUID="e362b475-7544-4095-a465-24905a71c6c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 15:30:21.867616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:21.858706 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" event={"ID":"4b57651a-2c8b-4df4-84c2-4f5fba809417","Type":"ContainerStarted","Data":"6400399bb8446061e1e35c6262cf1fac35d078822a9137dafa221a9207b30923"} Nov 03 15:30:21.936783 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:21.936686 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 15:30:21.989609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:21.970967 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:22.045632 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.026710 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" podUID="58f82529-9942-46d4-98dc-fa872122b973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 15:30:22.045632 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:21.993113 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:22.045632 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.028146 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" event={"ID":"d6d2a233-488a-42d3-8e7b-cd7868a6f167","Type":"ContainerStarted","Data":"7a59364f66c892b477fccc0ec046f6e1e7fc631c7e9ea39cd80d16a6fde532c7"} Nov 03 15:30:22.204662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.199058 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg"] Nov 03 15:30:22.312605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.278829 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:22.410256 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.410184 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" podStartSLOduration=5.410130876 podStartE2EDuration="5.410130876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:22.286232139 +0000 UTC m=+1332.129353682" watchObservedRunningTime="2026-11-03 15:30:22.410130876 +0000 UTC m=+1332.253252419" Nov 03 15:30:22.442622 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.437635 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be6660d4-9d6b-4706-9cfd-0d204e1b4e45" path="/var/lib/kubelet/pods/be6660d4-9d6b-4706-9cfd-0d204e1b4e45/volumes" Nov 03 15:30:22.442622 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.438306 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b8f36c-0d65-4523-b767-cd6970eb594e" path="/var/lib/kubelet/pods/c7b8f36c-0d65-4523-b767-cd6970eb594e/volumes" Nov 03 15:30:22.470663 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.461829 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f78fd6-29895330-vd4sg"] Nov 03 15:30:22.470663 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.461989 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:22.470663 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.462029 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:22.470663 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.468195 4557 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:22.470663 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.468250 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" podUID="d6d2a233-488a-42d3-8e7b-cd7868a6f167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 15:30:22.503606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.502738 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:22.576315 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.576270 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:22.577390 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.577365 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:30:22.590934 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.579933 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" podStartSLOduration=4.579915383 podStartE2EDuration="4.579915383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:22.442197349 +0000 UTC m=+1332.285318906" watchObservedRunningTime="2026-11-03 15:30:22.579915383 +0000 UTC m=+1332.423036926" Nov 03 15:30:22.681774 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.681718 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:22.682411 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.682377 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:22.682667 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.682648 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:30:22.766658 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.752681 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 15:30:22.770508 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.770473 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" podUID="3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 15:30:22.770758 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.768919 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:22.770858 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.770843 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" podUID="e362b475-7544-4095-a465-24905a71c6c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 15:30:22.904605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.902693 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:22.904605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.903115 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 03 15:30:22.904605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.903374 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 15:30:22.985606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:22.953165 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:23.046637 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.018776 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:23.047097 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.036662 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:23.047616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.046906 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 15:30:23.047797 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.046955 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" podUID="58f82529-9942-46d4-98dc-fa872122b973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 15:30:23.048040 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.048021 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:30:23.149717 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.078617 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:23.348475 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.348377 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:30:23.368030 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.367997 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:23.376410 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.376367 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:23.491376 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.475619 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:30:23.491376 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.475743 4557 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:23.491376 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.475786 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" podUID="d6d2a233-488a-42d3-8e7b-cd7868a6f167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 15:30:23.491376 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.475985 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:23.491376 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.484659 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:23.502964 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.502920 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:23.503648 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.503624 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:30:23.595985 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.595942 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 15:30:23.607136 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.606708 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:23.693611 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.693031 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 15:30:23.693611 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.693092 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" podUID="3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 15:30:23.834633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.809513 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:23.834633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.809572 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" podUID="e362b475-7544-4095-a465-24905a71c6c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 15:30:23.950771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.927486 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:23.950771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.927866 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Nov 03 15:30:23.950771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.928113 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 03 15:30:23.951377 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.951306 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:30:23.951572 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.951525 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:30:23.951757 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.951705 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:30:24.006515 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:23.974399 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:24.090897 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.049837 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:24.095505 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.049871 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 15:30:24.095769 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.049909 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:24.096203 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.080748 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:24.096568 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.090756 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" podUID="58f82529-9942-46d4-98dc-fa872122b973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 15:30:24.096860 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.096841 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:30:24.097015 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.097002 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:30:24.254009 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.253920 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:30:24.449874 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.449838 4557 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:24.450131 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.450102 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" podUID="d6d2a233-488a-42d3-8e7b-cd7868a6f167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 15:30:24.544770 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.518943 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 03 15:30:24.564626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.559694 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:24.633613 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.602884 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:24.674607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.650189 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 03 15:30:24.714610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.697628 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 15:30:24.714610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.697690 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" podUID="3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 15:30:24.714610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.713552 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="192929b4-7ed3-4e9d-9a32-f18088dc5910" path="/var/lib/kubelet/pods/192929b4-7ed3-4e9d-9a32-f18088dc5910/volumes" Nov 03 15:30:24.714610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.714150 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:30:24.714610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.714203 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:30:24.745608 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.744284 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:24.745608 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:24.744346 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" podUID="e362b475-7544-4095-a465-24905a71c6c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 15:30:25.086305 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.086261 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:25.108075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.108022 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:25.124612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.108759 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 15:30:25.144605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.139139 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 15:30:25.167373 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.148059 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:25.179418 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.179375 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded" Nov 03 15:30:25.197055 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.197013 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:25.224606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.199223 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:30:25.495660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.249151 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:25.534608 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.533672 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:25.559604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.272624 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:25.559604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.540786 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" podUID="58f82529-9942-46d4-98dc-fa872122b973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 15:30:25.559604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.435787 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:25.559604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.435718 4557 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:25.559604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.547357 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" podUID="d6d2a233-488a-42d3-8e7b-cd7868a6f167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 15:30:25.559604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.251264 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 03 15:30:25.560366 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.560219 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:25.643609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.640021 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:25.663606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.659677 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:25.721291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.706557 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:25.721620 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.721563 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" podUID="e362b475-7544-4095-a465-24905a71c6c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 15:30:25.725773 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.713690 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 15:30:25.725962 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.725937 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" podUID="3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 15:30:25.787066 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.786973 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:26.008606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:25.983149 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:26.009417 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.009375 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" podUID="58f82529-9942-46d4-98dc-fa872122b973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 15:30:26.010760 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.010718 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:26.185607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.182094 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 15:30:26.190300 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.190264 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:26.197877 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.197824 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 15:30:26.209787 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.209751 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:26.210196 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.210171 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:26.261601 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.245708 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:26.262043 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.250728 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:26.346969 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.346937 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:30:26.487184 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.463742 4557 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:26.487184 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.463800 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" podUID="d6d2a233-488a-42d3-8e7b-cd7868a6f167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 15:30:26.539608 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.509663 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Nov 03 15:30:26.603753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.603696 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:26.609748 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.609701 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:26.624726 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.624681 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tr6nx" podUID="7f27f30d-07d4-4ed3-b1db-4645289d94fe" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:26.641108 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.641061 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 15:30:26.641887 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.641862 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:30:26.686358 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.686320 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk"] Nov 03 15:30:26.771255 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.687667 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:26.771255 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.757477 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 15:30:26.771255 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.762969 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" podUID="3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 15:30:26.771255 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.759537 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:26.771255 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.763023 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" podUID="e362b475-7544-4095-a465-24905a71c6c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 15:30:26.870304 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.870271 4557 cpu_manager.go:414] "containerMap: removing container" podUID="192929b4-7ed3-4e9d-9a32-f18088dc5910" containerName="cleanup" Nov 03 15:30:26.905754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.905719 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192929b4-7ed3-4e9d-9a32-f18088dc5910" containerName="cleanup" Nov 03 15:30:26.905992 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.905975 4557 cpu_manager.go:414] "containerMap: removing container" podUID="5a8f0988-165b-4bee-85c2-357bfb4e4a60" containerName="revoke-keys" Nov 03 15:30:26.906119 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.906106 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a8f0988-165b-4bee-85c2-357bfb4e4a60" containerName="revoke-keys" Nov 03 15:30:26.906226 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.906216 4557 cpu_manager.go:414] "containerMap: removing container" podUID="c7b8f36c-0d65-4523-b767-cd6970eb594e" containerName="cleanup" Nov 03 15:30:26.906301 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.906290 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b8f36c-0d65-4523-b767-cd6970eb594e" containerName="cleanup" Nov 03 15:30:26.906386 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.906377 4557 cpu_manager.go:414] "containerMap: removing container" podUID="c7b8f36c-0d65-4523-b767-cd6970eb594e" containerName="cleanup" Nov 03 15:30:26.906456 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.906447 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b8f36c-0d65-4523-b767-cd6970eb594e" containerName="cleanup" Nov 03 15:30:26.906557 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.906547 4557 cpu_manager.go:414] "containerMap: removing container" podUID="192929b4-7ed3-4e9d-9a32-f18088dc5910" containerName="cleanup" Nov 03 15:30:26.906652 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.906640 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192929b4-7ed3-4e9d-9a32-f18088dc5910" containerName="cleanup" Nov 03 15:30:26.906932 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.906915 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b8f36c-0d65-4523-b767-cd6970eb594e" containerName="cleanup" Nov 03 15:30:26.907057 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.907045 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="192929b4-7ed3-4e9d-9a32-f18088dc5910" containerName="cleanup" Nov 03 15:30:26.907143 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.907131 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="192929b4-7ed3-4e9d-9a32-f18088dc5910" containerName="cleanup" Nov 03 15:30:26.907221 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.907212 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="192929b4-7ed3-4e9d-9a32-f18088dc5910" containerName="cleanup" Nov 03 15:30:26.907300 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.907291 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a8f0988-165b-4bee-85c2-357bfb4e4a60" containerName="revoke-keys" Nov 03 15:30:26.907375 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.907366 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b8f36c-0d65-4523-b767-cd6970eb594e" containerName="cleanup" Nov 03 15:30:26.955978 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.955943 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:26.956239 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:26.956209 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" podUID="58f82529-9942-46d4-98dc-fa872122b973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 15:30:27.099618 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.097767 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded" Nov 03 15:30:27.099618 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.099147 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:30:27.099618 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.099275 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk"] Nov 03 15:30:27.104612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.099886 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk" Nov 03 15:30:27.158967 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.139665 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e9334a6-331a-4402-b566-0ff28d91dd5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13702e89-cz2xk\" (UID: \"3e9334a6-331a-4402-b566-0ff28d91dd5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk" Nov 03 15:30:27.158967 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.139717 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e9334a6-331a-4402-b566-0ff28d91dd5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13702e89-cz2xk\" (UID: \"3e9334a6-331a-4402-b566-0ff28d91dd5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk" Nov 03 15:30:27.158967 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.139791 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnhxb\" (UniqueName: \"kubernetes.io/projected/3e9334a6-331a-4402-b566-0ff28d91dd5e-kube-api-access-qnhxb\") pod \"maas-api-revoke-sub-13702e89-cz2xk\" (UID: \"3e9334a6-331a-4402-b566-0ff28d91dd5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk" Nov 03 15:30:27.187605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.185716 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:27.213724 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.213668 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:27.265603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.254497 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e9334a6-331a-4402-b566-0ff28d91dd5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13702e89-cz2xk\" (UID: \"3e9334a6-331a-4402-b566-0ff28d91dd5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk" Nov 03 15:30:27.265603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.254555 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e9334a6-331a-4402-b566-0ff28d91dd5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13702e89-cz2xk\" (UID: \"3e9334a6-331a-4402-b566-0ff28d91dd5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk" Nov 03 15:30:27.265603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.254672 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnhxb\" (UniqueName: \"kubernetes.io/projected/3e9334a6-331a-4402-b566-0ff28d91dd5e-kube-api-access-qnhxb\") pod \"maas-api-revoke-sub-13702e89-cz2xk\" (UID: \"3e9334a6-331a-4402-b566-0ff28d91dd5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk" Nov 03 15:30:27.265603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.262545 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e9334a6-331a-4402-b566-0ff28d91dd5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13702e89-cz2xk\" (UID: \"3e9334a6-331a-4402-b566-0ff28d91dd5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk" Nov 03 15:30:27.295114 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.295062 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:30:27.295374 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.295346 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:30:27.303139 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.303112 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:30:27.330381 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.330315 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e9334a6-331a-4402-b566-0ff28d91dd5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13702e89-cz2xk\" (UID: \"3e9334a6-331a-4402-b566-0ff28d91dd5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk" Nov 03 15:30:27.343370 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.333173 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnhxb\" (UniqueName: \"kubernetes.io/projected/3e9334a6-331a-4402-b566-0ff28d91dd5e-kube-api-access-qnhxb\") pod \"maas-api-revoke-sub-13702e89-cz2xk\" (UID: \"3e9334a6-331a-4402-b566-0ff28d91dd5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk" Nov 03 15:30:27.421609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.421435 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:27.437466 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.423295 4557 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:27.437748 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.437713 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" podUID="d6d2a233-488a-42d3-8e7b-cd7868a6f167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 15:30:27.507053 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.505802 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk" Nov 03 15:30:27.510824 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.510112 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:27.616617 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.612695 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:27.667632 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.646152 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 15:30:27.697864 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.696173 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:30:27.767261 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.733269 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 15:30:27.767261 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.733334 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" podUID="3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 15:30:27.767261 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.733977 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:30:27.779621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.767658 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:30:27.800680 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.781757 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:27.800680 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.781810 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" podUID="e362b475-7544-4095-a465-24905a71c6c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 15:30:27.916700 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.915698 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:27.980638 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.978879 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:27.980638 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:27.978941 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" podUID="58f82529-9942-46d4-98dc-fa872122b973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 15:30:28.185623 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:28.181918 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:30:28.526638 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:28.520484 4557 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:28.526638 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:28.520547 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" podUID="d6d2a233-488a-42d3-8e7b-cd7868a6f167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 15:30:28.526638 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:28.520643 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:28.623781 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:28.623680 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:28.799665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:28.793385 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:28.799665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:28.794320 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:28.799665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:28.796562 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:30:28.824616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:28.801748 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:28.824616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:28.805688 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 15:30:28.824616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:28.805729 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" podUID="e362b475-7544-4095-a465-24905a71c6c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 15:30:28.941115 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:28.941088 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:30:28.952606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:28.949496 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk"] Nov 03 15:30:28.968177 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:28.968131 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 15:30:28.968441 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:28.968398 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" podUID="58f82529-9942-46d4-98dc-fa872122b973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 15:30:29.219437 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:29.219343 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:29.223227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:29.223192 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:29.235927 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:29.235904 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:30:29.319058 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:29.319010 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4"] Nov 03 15:30:29.319530 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:29.319494 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" podUID="72286ed3-dafb-4c80-b6bf-6db70a73f2a6" containerName="istio-proxy" containerID="cri-o://6a6c373a0fe1db401066e573fb433fa979eb8ae7eaaa64440cc37272d45face5" gracePeriod=30 Nov 03 15:30:29.328426 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:29.328399 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk" event={"ID":"3e9334a6-331a-4402-b566-0ff28d91dd5e","Type":"ContainerStarted","Data":"7f90ab4d039bdcaa658ada51eb86ebb081feb540aad5cb2d42809128bf562c48"} Nov 03 15:30:29.424033 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:29.423994 4557 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 15:30:29.424197 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:29.424055 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" podUID="d6d2a233-488a-42d3-8e7b-cd7868a6f167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 15:30:29.715625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:29.710286 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:29.800252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:29.800216 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:29.801197 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:29.801178 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:29.803439 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:29.803382 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:30:29.991981 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:29.991794 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:29.992834 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:29.992802 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:29.993736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:29.993717 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:30:30.053214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:30.053111 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc"] Nov 03 15:30:30.063815 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:30.063787 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eae949b-97zlc"] Nov 03 15:30:30.289757 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:30.289715 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10fcf8a3-543c-4728-bc62-26a2385561a4" path="/var/lib/kubelet/pods/10fcf8a3-543c-4728-bc62-26a2385561a4/volumes" Nov 03 15:30:30.338649 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:30.338622 4557 generic.go:356] "Generic (PLEG): container finished" podID="3e9334a6-331a-4402-b566-0ff28d91dd5e" containerID="739c5ef391738f1ba58c771d66f9e5f08107c39e67b3d6f2ad413c865a5da2c1" exitCode=0 Nov 03 15:30:30.338781 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:30.338746 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk" event={"ID":"3e9334a6-331a-4402-b566-0ff28d91dd5e","Type":"ContainerDied","Data":"739c5ef391738f1ba58c771d66f9e5f08107c39e67b3d6f2ad413c865a5da2c1"} Nov 03 15:30:30.873688 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:30.873649 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:30.874121 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:30.874080 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:30.874924 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:30.874902 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:30:31.515037 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:31.515012 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk" Nov 03 15:30:31.601911 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:31.601882 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e9334a6-331a-4402-b566-0ff28d91dd5e-kube-api-access-qnhxb\" (UniqueName: \"kubernetes.io/projected/3e9334a6-331a-4402-b566-0ff28d91dd5e-kube-api-access-qnhxb\") pod \"3e9334a6-331a-4402-b566-0ff28d91dd5e\" (UID: \"3e9334a6-331a-4402-b566-0ff28d91dd5e\") " Nov 03 15:30:31.602054 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:31.601947 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e9334a6-331a-4402-b566-0ff28d91dd5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e9334a6-331a-4402-b566-0ff28d91dd5e-maas-api-xks-ca\") pod \"3e9334a6-331a-4402-b566-0ff28d91dd5e\" (UID: \"3e9334a6-331a-4402-b566-0ff28d91dd5e\") " Nov 03 15:30:31.602224 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:31.602200 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e9334a6-331a-4402-b566-0ff28d91dd5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e9334a6-331a-4402-b566-0ff28d91dd5e-maas-api-openshift-ca\") pod \"3e9334a6-331a-4402-b566-0ff28d91dd5e\" (UID: \"3e9334a6-331a-4402-b566-0ff28d91dd5e\") " Nov 03 15:30:31.602811 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:31.602557 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9334a6-331a-4402-b566-0ff28d91dd5e-maas-api-openshift-ca" pod "3e9334a6-331a-4402-b566-0ff28d91dd5e" (UID: "3e9334a6-331a-4402-b566-0ff28d91dd5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:31.602894 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:31.602854 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e9334a6-331a-4402-b566-0ff28d91dd5e-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:31.604274 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:31.604248 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9334a6-331a-4402-b566-0ff28d91dd5e-kube-api-access-qnhxb" pod "3e9334a6-331a-4402-b566-0ff28d91dd5e" (UID: "3e9334a6-331a-4402-b566-0ff28d91dd5e"). InnerVolumeSpecName "kube-api-access-qnhxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:31.604391 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:31.604374 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e9334a6-331a-4402-b566-0ff28d91dd5e-maas-api-xks-ca" pod "3e9334a6-331a-4402-b566-0ff28d91dd5e" (UID: "3e9334a6-331a-4402-b566-0ff28d91dd5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:30:31.704770 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:31.704727 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnhxb\" (UniqueName: \"kubernetes.io/projected/3e9334a6-331a-4402-b566-0ff28d91dd5e-kube-api-access-qnhxb\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:31.704770 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:31.704764 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e9334a6-331a-4402-b566-0ff28d91dd5e-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:32.495168 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:32.495139 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk" Nov 03 15:30:32.496609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:32.496566 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk" event={"ID":"3e9334a6-331a-4402-b566-0ff28d91dd5e","Type":"ContainerDied","Data":"7f90ab4d039bdcaa658ada51eb86ebb081feb540aad5cb2d42809128bf562c48"} Nov 03 15:30:32.497311 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:32.497290 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f90ab4d039bdcaa658ada51eb86ebb081feb540aad5cb2d42809128bf562c48" Nov 03 15:30:32.624911 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:32.624838 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:32.935605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:32.934254 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:34.159606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:34.158771 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:34.203871 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:34.203822 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r"] Nov 03 15:30:34.239607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:34.219818 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:34.289604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:34.288662 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:34.305608 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:34.303691 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:34.435569 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:34.435531 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Nov 03 15:30:34.501667 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:34.501631 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5789edb-kpn7r"] Nov 03 15:30:34.502203 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:34.502156 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Nov 03 15:30:34.546749 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:34.546701 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 03 15:30:34.847851 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:34.833174 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 15:30:34.875621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:34.849650 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:34.875621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:34.849719 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 03 15:30:35.127726 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.127664 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:35.155606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.140117 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:35.205750 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.195721 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:35.205750 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.196165 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:30:35.240625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.239492 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:35.240625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.239939 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:30:35.392662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.391735 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:35.416501 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.393250 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:30:35.416501 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.393952 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 03 15:30:35.416501 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.394242 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:30:35.531732 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.531675 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:35.557606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.532355 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:30:35.561603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.558435 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:35.562130 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.562108 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:30:35.564103 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.564076 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:35.564537 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.564518 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:30:35.632095 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.632048 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:35.752213 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.752107 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 15:30:35.898297 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.898245 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 15:30:35.930608 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.906109 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:30:35.970607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.932704 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:35.997058 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.956681 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Nov 03 15:30:36.027664 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:35.999170 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:30:36.028225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:36.028199 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:30:36.240720 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:36.237741 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:36.240720 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:36.237856 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 15:30:36.242467 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:36.242439 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:30:36.453999 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:36.446291 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 15:30:36.580606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:36.579791 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 03 15:30:36.587045 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:36.581170 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Nov 03 15:30:36.587045 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:36.581503 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:36.659605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:36.590708 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:30:36.659605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:36.590774 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" podUID="f7bb3b78-db67-4de5-9b60-10d26274a199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:36.688607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:36.660391 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:36.817604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:36.781853 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:36.817604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:36.800376 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 15:30:36.827293 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:36.827253 4557 generic.go:356] "Generic (PLEG): container finished" podID="72286ed3-dafb-4c80-b6bf-6db70a73f2a6" containerID="6a6c373a0fe1db401066e573fb433fa979eb8ae7eaaa64440cc37272d45face5" exitCode=0 Nov 03 15:30:36.845729 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:36.841876 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:36.900768 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:36.900711 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:37.035603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.023355 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:37.176607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.058369 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:37.254639 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.247365 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d31fb2e6-c807-45ca-8800-9513f0a5b8e1" path="/var/lib/kubelet/pods/d31fb2e6-c807-45ca-8800-9513f0a5b8e1/volumes" Nov 03 15:30:37.254639 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.247913 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" event={"ID":"72286ed3-dafb-4c80-b6bf-6db70a73f2a6","Type":"ContainerDied","Data":"6a6c373a0fe1db401066e573fb433fa979eb8ae7eaaa64440cc37272d45face5"} Nov 03 15:30:37.254639 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.248049 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:30:37.254639 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.248117 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:30:37.258602 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.256110 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:37.453663 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.453462 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:37.527603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.527555 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:30:37.563608 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.541101 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-kube-api-access-rh8x9\" (UniqueName: \"kubernetes.io/projected/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-kube-api-access-rh8x9\") pod \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " Nov 03 15:30:37.592737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.592706 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istiod-ca-cert\") pod \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " Nov 03 15:30:37.596611 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.593008 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-workload-socket\") pod \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " Nov 03 15:30:37.604855 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.604824 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-envoy\") pod \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " Nov 03 15:30:37.605131 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.605113 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-credential-socket\") pod \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " Nov 03 15:30:37.610693 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.605340 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-token\" (UniqueName: \"kubernetes.io/projected/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-token\") pod \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " Nov 03 15:30:37.634971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.634939 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-data\") pod \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " Nov 03 15:30:37.635362 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.635341 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-podinfo\") pod \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " Nov 03 15:30:37.635507 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.635491 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-workload-certs\") pod \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\" (UID: \"72286ed3-dafb-4c80-b6bf-6db70a73f2a6\") " Nov 03 15:30:37.694104 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.590892 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:37.695264 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.590912 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Nov 03 15:30:37.696048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.563905 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-zgkl4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:30:37.696048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.695461 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" podUID="72286ed3-dafb-4c80-b6bf-6db70a73f2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:37.696048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.590819 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:37.696048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.590867 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 03 15:30:37.697529 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.613150 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-workload-socket" pod "72286ed3-dafb-4c80-b6bf-6db70a73f2a6" (UID: "72286ed3-dafb-4c80-b6bf-6db70a73f2a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:30:37.697529 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.613576 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-credential-socket" pod "72286ed3-dafb-4c80-b6bf-6db70a73f2a6" (UID: "72286ed3-dafb-4c80-b6bf-6db70a73f2a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:30:37.697529 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.634786 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istiod-ca-cert" pod "72286ed3-dafb-4c80-b6bf-6db70a73f2a6" (UID: "72286ed3-dafb-4c80-b6bf-6db70a73f2a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:37.697529 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.644447 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-kube-api-access-rh8x9" pod "72286ed3-dafb-4c80-b6bf-6db70a73f2a6" (UID: "72286ed3-dafb-4c80-b6bf-6db70a73f2a6"). InnerVolumeSpecName "kube-api-access-rh8x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:37.697529 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.644853 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-data" pod "72286ed3-dafb-4c80-b6bf-6db70a73f2a6" (UID: "72286ed3-dafb-4c80-b6bf-6db70a73f2a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:30:37.697529 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.647633 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-workload-certs" pod "72286ed3-dafb-4c80-b6bf-6db70a73f2a6" (UID: "72286ed3-dafb-4c80-b6bf-6db70a73f2a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:30:37.697529 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.647717 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-podinfo" pod "72286ed3-dafb-4c80-b6bf-6db70a73f2a6" (UID: "72286ed3-dafb-4c80-b6bf-6db70a73f2a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:30:37.803229 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.803183 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 15:30:37.823186 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.803475 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:37.824076 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.803512 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 15:30:37.859660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.805280 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:37.859660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.836803 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:37.859660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.836830 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh8x9\" (UniqueName: \"kubernetes.io/projected/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-kube-api-access-rh8x9\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:37.859660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.836849 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:37.859660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.836865 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:37.859660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.836880 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:37.859660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.836897 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:37.891758 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.861339 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:37.891758 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.861454 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:37.898613 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.893040 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:37.963371 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.899106 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-token" pod "72286ed3-dafb-4c80-b6bf-6db70a73f2a6" (UID: "72286ed3-dafb-4c80-b6bf-6db70a73f2a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:37.963371 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.924873 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-envoy" pod "72286ed3-dafb-4c80-b6bf-6db70a73f2a6" (UID: "72286ed3-dafb-4c80-b6bf-6db70a73f2a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:30:37.963371 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:37.914856 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 15:30:38.041533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.041499 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:38.057518 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.057439 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72286ed3-dafb-4c80-b6bf-6db70a73f2a6-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:38.099603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.099557 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:30:38.100827 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.100794 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Nov 03 15:30:38.104204 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.104182 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" Nov 03 15:30:38.144699 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.142479 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4" event={"ID":"72286ed3-dafb-4c80-b6bf-6db70a73f2a6","Type":"ContainerDied","Data":"fe7737f9095c6c20bc5b210f665fbeb93b2fcbae999082202eab0ba1c3c1c99d"} Nov 03 15:30:38.144699 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.142607 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:30:38.144699 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.143479 4557 scope.go:122] "RemoveContainer" containerID="6a6c373a0fe1db401066e573fb433fa979eb8ae7eaaa64440cc37272d45face5" Nov 03 15:30:38.189603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.189555 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:38.327660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.302489 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 15:30:38.351625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.346704 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:30:38.351625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.346768 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" podUID="18e9b467-83f6-4526-b8ae-2723410e32a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:38.472831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.470226 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:38.556779 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.548631 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:38.623671 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.610163 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:30:38.693005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.692954 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded" Nov 03 15:30:38.693327 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.693301 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Nov 03 15:30:38.711665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.710957 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:38.825434 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.825377 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 15:30:38.831868 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.831546 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:38.843476 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.840524 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:30:38.843966 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.829473 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:38.851723 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.851695 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:30:38.852093 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.852070 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:30:38.864666 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.852862 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:30:38.864666 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.858361 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:30:38.865022 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.865008 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:30:38.868298 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.868284 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:30:38.884325 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.884302 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:30:38.884492 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:30:38.884363 4557 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72286ed3_dafb_4c80_b6bf_6db70a73f2a6.slice\": RecentStats: unable to find data in memory cache]" Nov 03 15:30:38.885184 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.885166 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:30:38.903442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.903410 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4"] Nov 03 15:30:38.915059 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:38.915033 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-zgkl4"] Nov 03 15:30:40.287189 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:40.287153 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72286ed3-dafb-4c80-b6bf-6db70a73f2a6" path="/var/lib/kubelet/pods/72286ed3-dafb-4c80-b6bf-6db70a73f2a6/volumes" Nov 03 15:30:41.654660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:41.654627 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:41.865741 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:41.865705 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:42.712659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.712631 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb"] Nov 03 15:30:42.713175 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.713161 4557 cpu_manager.go:414] "containerMap: removing container" podUID="192929b4-7ed3-4e9d-9a32-f18088dc5910" containerName="cleanup" Nov 03 15:30:42.713220 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.713176 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192929b4-7ed3-4e9d-9a32-f18088dc5910" containerName="cleanup" Nov 03 15:30:42.713220 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.713189 4557 cpu_manager.go:414] "containerMap: removing container" podUID="72286ed3-dafb-4c80-b6bf-6db70a73f2a6" containerName="istio-proxy" Nov 03 15:30:42.713220 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.713195 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72286ed3-dafb-4c80-b6bf-6db70a73f2a6" containerName="istio-proxy" Nov 03 15:30:42.713220 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.713214 4557 cpu_manager.go:414] "containerMap: removing container" podUID="3e9334a6-331a-4402-b566-0ff28d91dd5e" containerName="revoke-keys" Nov 03 15:30:42.713220 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.713218 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e9334a6-331a-4402-b566-0ff28d91dd5e" containerName="revoke-keys" Nov 03 15:30:42.713375 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.713228 4557 cpu_manager.go:414] "containerMap: removing container" podUID="c7b8f36c-0d65-4523-b767-cd6970eb594e" containerName="cleanup" Nov 03 15:30:42.713375 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.713233 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b8f36c-0d65-4523-b767-cd6970eb594e" containerName="cleanup" Nov 03 15:30:42.713375 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.713307 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="72286ed3-dafb-4c80-b6bf-6db70a73f2a6" containerName="istio-proxy" Nov 03 15:30:42.713375 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.713316 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e9334a6-331a-4402-b566-0ff28d91dd5e" containerName="revoke-keys" Nov 03 15:30:42.713375 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.713323 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b8f36c-0d65-4523-b767-cd6970eb594e" containerName="cleanup" Nov 03 15:30:42.718388 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.718372 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb" Nov 03 15:30:42.782815 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.782777 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb"] Nov 03 15:30:42.783677 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.783644 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f472ac9-570a-4124-ba65-e9f81b49342d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-189933f2-cbzvb\" (UID: \"4f472ac9-570a-4124-ba65-e9f81b49342d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb" Nov 03 15:30:42.783809 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.783692 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f472ac9-570a-4124-ba65-e9f81b49342d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-189933f2-cbzvb\" (UID: \"4f472ac9-570a-4124-ba65-e9f81b49342d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb" Nov 03 15:30:42.783885 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.783869 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcslr\" (UniqueName: \"kubernetes.io/projected/4f472ac9-570a-4124-ba65-e9f81b49342d-kube-api-access-jcslr\") pod \"maas-api-revoke-sub-189933f2-cbzvb\" (UID: \"4f472ac9-570a-4124-ba65-e9f81b49342d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb" Nov 03 15:30:42.808201 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.808177 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:30:42.808323 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.808234 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:30:42.885213 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.885172 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f472ac9-570a-4124-ba65-e9f81b49342d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-189933f2-cbzvb\" (UID: \"4f472ac9-570a-4124-ba65-e9f81b49342d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb" Nov 03 15:30:42.885405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.885226 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f472ac9-570a-4124-ba65-e9f81b49342d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-189933f2-cbzvb\" (UID: \"4f472ac9-570a-4124-ba65-e9f81b49342d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb" Nov 03 15:30:42.885405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.885276 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcslr\" (UniqueName: \"kubernetes.io/projected/4f472ac9-570a-4124-ba65-e9f81b49342d-kube-api-access-jcslr\") pod \"maas-api-revoke-sub-189933f2-cbzvb\" (UID: \"4f472ac9-570a-4124-ba65-e9f81b49342d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb" Nov 03 15:30:42.885921 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.885893 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f472ac9-570a-4124-ba65-e9f81b49342d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-189933f2-cbzvb\" (UID: \"4f472ac9-570a-4124-ba65-e9f81b49342d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb" Nov 03 15:30:42.887560 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.887540 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f472ac9-570a-4124-ba65-e9f81b49342d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-189933f2-cbzvb\" (UID: \"4f472ac9-570a-4124-ba65-e9f81b49342d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb" Nov 03 15:30:42.948743 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:42.948714 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcslr\" (UniqueName: \"kubernetes.io/projected/4f472ac9-570a-4124-ba65-e9f81b49342d-kube-api-access-jcslr\") pod \"maas-api-revoke-sub-189933f2-cbzvb\" (UID: \"4f472ac9-570a-4124-ba65-e9f81b49342d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb" Nov 03 15:30:43.032345 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:43.032270 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb" Nov 03 15:30:43.146590 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:43.146545 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh"] Nov 03 15:30:43.151862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:43.151836 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh" Nov 03 15:30:43.190322 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:43.190293 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442544d0-79c6-486e-a3bd-163752dc0112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b05dca9f-j47rh\" (UID: \"442544d0-79c6-486e-a3bd-163752dc0112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh" Nov 03 15:30:43.190480 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:43.190345 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442544d0-79c6-486e-a3bd-163752dc0112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b05dca9f-j47rh\" (UID: \"442544d0-79c6-486e-a3bd-163752dc0112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh" Nov 03 15:30:43.190551 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:43.190490 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsc7v\" (UniqueName: \"kubernetes.io/projected/442544d0-79c6-486e-a3bd-163752dc0112-kube-api-access-zsc7v\") pod \"maas-api-revoke-sub-b05dca9f-j47rh\" (UID: \"442544d0-79c6-486e-a3bd-163752dc0112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh" Nov 03 15:30:43.208904 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:43.208877 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh"] Nov 03 15:30:43.291741 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:43.291658 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442544d0-79c6-486e-a3bd-163752dc0112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b05dca9f-j47rh\" (UID: \"442544d0-79c6-486e-a3bd-163752dc0112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh" Nov 03 15:30:43.291741 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:43.291720 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442544d0-79c6-486e-a3bd-163752dc0112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b05dca9f-j47rh\" (UID: \"442544d0-79c6-486e-a3bd-163752dc0112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh" Nov 03 15:30:43.291969 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:43.291806 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsc7v\" (UniqueName: \"kubernetes.io/projected/442544d0-79c6-486e-a3bd-163752dc0112-kube-api-access-zsc7v\") pod \"maas-api-revoke-sub-b05dca9f-j47rh\" (UID: \"442544d0-79c6-486e-a3bd-163752dc0112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh" Nov 03 15:30:43.292981 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:43.292947 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442544d0-79c6-486e-a3bd-163752dc0112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b05dca9f-j47rh\" (UID: \"442544d0-79c6-486e-a3bd-163752dc0112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh" Nov 03 15:30:43.294903 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:43.294882 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442544d0-79c6-486e-a3bd-163752dc0112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b05dca9f-j47rh\" (UID: \"442544d0-79c6-486e-a3bd-163752dc0112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh" Nov 03 15:30:43.357200 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:43.357166 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsc7v\" (UniqueName: \"kubernetes.io/projected/442544d0-79c6-486e-a3bd-163752dc0112-kube-api-access-zsc7v\") pod \"maas-api-revoke-sub-b05dca9f-j47rh\" (UID: \"442544d0-79c6-486e-a3bd-163752dc0112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh" Nov 03 15:30:43.416994 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:43.416971 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb"] Nov 03 15:30:43.417107 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:30:43.416969 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f472ac9_570a_4124_ba65_e9f81b49342d.slice/crio-eb85eeaafe008e11aa932a77c5c54c3297cfedd449765bc8bb53fa2aff04d38f WatchSource:0}: Error finding container eb85eeaafe008e11aa932a77c5c54c3297cfedd449765bc8bb53fa2aff04d38f: Status 404 returned error can't find the container with id eb85eeaafe008e11aa932a77c5c54c3297cfedd449765bc8bb53fa2aff04d38f Nov 03 15:30:43.467557 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:43.467527 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh" Nov 03 15:30:43.958124 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:30:43.958079 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442544d0_79c6_486e_a3bd_163752dc0112.slice/crio-a81496845deb8a182c0f586d383086f4a4a1e528ef8a61d522c82da3b689adac WatchSource:0}: Error finding container a81496845deb8a182c0f586d383086f4a4a1e528ef8a61d522c82da3b689adac: Status 404 returned error can't find the container with id a81496845deb8a182c0f586d383086f4a4a1e528ef8a61d522c82da3b689adac Nov 03 15:30:43.958892 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:43.958864 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh"] Nov 03 15:30:44.167863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:44.167762 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh" event={"ID":"442544d0-79c6-486e-a3bd-163752dc0112","Type":"ContainerStarted","Data":"a81496845deb8a182c0f586d383086f4a4a1e528ef8a61d522c82da3b689adac"} Nov 03 15:30:44.169543 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:44.169506 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb" event={"ID":"4f472ac9-570a-4124-ba65-e9f81b49342d","Type":"ContainerStarted","Data":"1fc87ef25d1448d7e55c1e801b6de90c52c64b699df93597531db2ac2e864b69"} Nov 03 15:30:44.169735 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:44.169561 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb" event={"ID":"4f472ac9-570a-4124-ba65-e9f81b49342d","Type":"ContainerStarted","Data":"eb85eeaafe008e11aa932a77c5c54c3297cfedd449765bc8bb53fa2aff04d38f"} Nov 03 15:30:44.649131 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:44.649093 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:45.181376 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:45.181344 4557 generic.go:356] "Generic (PLEG): container finished" podID="442544d0-79c6-486e-a3bd-163752dc0112" containerID="106db7ee9c7f143a88b3b1a64621d0dc006883378eb89da47e7fe3304ca7df4b" exitCode=0 Nov 03 15:30:45.181813 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:45.181385 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh" event={"ID":"442544d0-79c6-486e-a3bd-163752dc0112","Type":"ContainerDied","Data":"106db7ee9c7f143a88b3b1a64621d0dc006883378eb89da47e7fe3304ca7df4b"} Nov 03 15:30:45.183129 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:45.183102 4557 generic.go:356] "Generic (PLEG): container finished" podID="4f472ac9-570a-4124-ba65-e9f81b49342d" containerID="1fc87ef25d1448d7e55c1e801b6de90c52c64b699df93597531db2ac2e864b69" exitCode=0 Nov 03 15:30:45.183213 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:45.183184 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb" event={"ID":"4f472ac9-570a-4124-ba65-e9f81b49342d","Type":"ContainerDied","Data":"1fc87ef25d1448d7e55c1e801b6de90c52c64b699df93597531db2ac2e864b69"} Nov 03 15:30:45.298286 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:45.298234 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb" podStartSLOduration=3.298216357 podStartE2EDuration="3.298216357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:44.286550259 +0000 UTC m=+1354.129671801" watchObservedRunningTime="2026-11-03 15:30:45.298216357 +0000 UTC m=+1355.141337900" Nov 03 15:30:45.516324 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:45.516290 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:45.753313 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:45.753220 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:46.361183 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.361157 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc"] Nov 03 15:30:46.365867 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.365843 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.409007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.408984 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb" Nov 03 15:30:46.463171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.445541 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f472ac9-570a-4124-ba65-e9f81b49342d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f472ac9-570a-4124-ba65-e9f81b49342d-maas-api-xks-ca\") pod \"4f472ac9-570a-4124-ba65-e9f81b49342d\" (UID: \"4f472ac9-570a-4124-ba65-e9f81b49342d\") " Nov 03 15:30:46.463171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.445628 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f472ac9-570a-4124-ba65-e9f81b49342d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f472ac9-570a-4124-ba65-e9f81b49342d-maas-api-openshift-ca\") pod \"4f472ac9-570a-4124-ba65-e9f81b49342d\" (UID: \"4f472ac9-570a-4124-ba65-e9f81b49342d\") " Nov 03 15:30:46.463171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.445763 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f472ac9-570a-4124-ba65-e9f81b49342d-kube-api-access-jcslr\" (UniqueName: \"kubernetes.io/projected/4f472ac9-570a-4124-ba65-e9f81b49342d-kube-api-access-jcslr\") pod \"4f472ac9-570a-4124-ba65-e9f81b49342d\" (UID: \"4f472ac9-570a-4124-ba65-e9f81b49342d\") " Nov 03 15:30:46.463171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.446090 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f472ac9-570a-4124-ba65-e9f81b49342d-maas-api-openshift-ca" pod "4f472ac9-570a-4124-ba65-e9f81b49342d" (UID: "4f472ac9-570a-4124-ba65-e9f81b49342d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:46.463171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.446496 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvkql\" (UniqueName: \"kubernetes.io/projected/932d5d02-ff9f-40e8-bc1e-7f672c989547-kube-api-access-nvkql\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.463171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.446569 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.463171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.446673 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.463171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.446761 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.463171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.446875 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.463171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.446934 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.463171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.446982 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.463171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.447217 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932d5d02-ff9f-40e8-bc1e-7f672c989547-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.463171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.447281 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.463171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.447416 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f472ac9-570a-4124-ba65-e9f81b49342d-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:46.463171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.448819 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f472ac9-570a-4124-ba65-e9f81b49342d-kube-api-access-jcslr" pod "4f472ac9-570a-4124-ba65-e9f81b49342d" (UID: "4f472ac9-570a-4124-ba65-e9f81b49342d"). InnerVolumeSpecName "kube-api-access-jcslr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:46.463171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.453137 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f472ac9-570a-4124-ba65-e9f81b49342d-maas-api-xks-ca" pod "4f472ac9-570a-4124-ba65-e9f81b49342d" (UID: "4f472ac9-570a-4124-ba65-e9f81b49342d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:30:46.463171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.455906 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh" Nov 03 15:30:46.476008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.475939 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc"] Nov 03 15:30:46.531250 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:30:46.531214 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:30:46.548240 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.548216 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/442544d0-79c6-486e-a3bd-163752dc0112-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442544d0-79c6-486e-a3bd-163752dc0112-maas-api-xks-ca\") pod \"442544d0-79c6-486e-a3bd-163752dc0112\" (UID: \"442544d0-79c6-486e-a3bd-163752dc0112\") " Nov 03 15:30:46.548378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.548350 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/442544d0-79c6-486e-a3bd-163752dc0112-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442544d0-79c6-486e-a3bd-163752dc0112-maas-api-openshift-ca\") pod \"442544d0-79c6-486e-a3bd-163752dc0112\" (UID: \"442544d0-79c6-486e-a3bd-163752dc0112\") " Nov 03 15:30:46.548499 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.548487 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/442544d0-79c6-486e-a3bd-163752dc0112-kube-api-access-zsc7v\" (UniqueName: \"kubernetes.io/projected/442544d0-79c6-486e-a3bd-163752dc0112-kube-api-access-zsc7v\") pod \"442544d0-79c6-486e-a3bd-163752dc0112\" (UID: \"442544d0-79c6-486e-a3bd-163752dc0112\") " Nov 03 15:30:46.548670 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.548651 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932d5d02-ff9f-40e8-bc1e-7f672c989547-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.548752 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.548690 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.548752 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.548704 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/442544d0-79c6-486e-a3bd-163752dc0112-maas-api-openshift-ca" pod "442544d0-79c6-486e-a3bd-163752dc0112" (UID: "442544d0-79c6-486e-a3bd-163752dc0112"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:46.548855 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.548753 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvkql\" (UniqueName: \"kubernetes.io/projected/932d5d02-ff9f-40e8-bc1e-7f672c989547-kube-api-access-nvkql\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.548855 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.548799 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.548961 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.548867 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.548961 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.548946 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.549076 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.548997 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.549076 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.549033 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.549076 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.549063 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.549243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.549055 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.549358 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.549333 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcslr\" (UniqueName: \"kubernetes.io/projected/4f472ac9-570a-4124-ba65-e9f81b49342d-kube-api-access-jcslr\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:46.549422 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.549368 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442544d0-79c6-486e-a3bd-163752dc0112-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:46.549422 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.549384 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f472ac9-570a-4124-ba65-e9f81b49342d-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:46.549422 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.549382 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.549603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.549427 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932d5d02-ff9f-40e8-bc1e-7f672c989547-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.549718 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.549692 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.550080 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.550050 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.550614 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.550554 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/442544d0-79c6-486e-a3bd-163752dc0112-maas-api-xks-ca" pod "442544d0-79c6-486e-a3bd-163752dc0112" (UID: "442544d0-79c6-486e-a3bd-163752dc0112"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:30:46.550957 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.550937 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442544d0-79c6-486e-a3bd-163752dc0112-kube-api-access-zsc7v" pod "442544d0-79c6-486e-a3bd-163752dc0112" (UID: "442544d0-79c6-486e-a3bd-163752dc0112"). InnerVolumeSpecName "kube-api-access-zsc7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:46.551321 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.551299 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.551719 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.551697 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.612919 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.612884 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.613081 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.613006 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvkql\" (UniqueName: \"kubernetes.io/projected/932d5d02-ff9f-40e8-bc1e-7f672c989547-kube-api-access-nvkql\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:46.650238 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.650214 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsc7v\" (UniqueName: \"kubernetes.io/projected/442544d0-79c6-486e-a3bd-163752dc0112-kube-api-access-zsc7v\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:46.650238 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.650238 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442544d0-79c6-486e-a3bd-163752dc0112-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:46.746494 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:46.746421 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:47.179955 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:30:47.179929 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932d5d02_ff9f_40e8_bc1e_7f672c989547.slice/crio-372bd0761f7cb12e749640c501e264fbe269a057bbcf5edf9d690b6e9ca7ae90 WatchSource:0}: Error finding container 372bd0761f7cb12e749640c501e264fbe269a057bbcf5edf9d690b6e9ca7ae90: Status 404 returned error can't find the container with id 372bd0761f7cb12e749640c501e264fbe269a057bbcf5edf9d690b6e9ca7ae90 Nov 03 15:30:47.182532 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:47.182500 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:47.182643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:47.182600 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:47.182686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:47.182641 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:30:47.183290 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:47.183266 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc"] Nov 03 15:30:47.218484 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:47.218461 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" event={"ID":"932d5d02-ff9f-40e8-bc1e-7f672c989547","Type":"ContainerStarted","Data":"372bd0761f7cb12e749640c501e264fbe269a057bbcf5edf9d690b6e9ca7ae90"} Nov 03 15:30:47.219970 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:47.219944 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh" event={"ID":"442544d0-79c6-486e-a3bd-163752dc0112","Type":"ContainerDied","Data":"a81496845deb8a182c0f586d383086f4a4a1e528ef8a61d522c82da3b689adac"} Nov 03 15:30:47.220073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:47.219991 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a81496845deb8a182c0f586d383086f4a4a1e528ef8a61d522c82da3b689adac" Nov 03 15:30:47.220073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:47.219954 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh" Nov 03 15:30:47.221466 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:47.221447 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb" event={"ID":"4f472ac9-570a-4124-ba65-e9f81b49342d","Type":"ContainerDied","Data":"eb85eeaafe008e11aa932a77c5c54c3297cfedd449765bc8bb53fa2aff04d38f"} Nov 03 15:30:47.221570 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:47.221479 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb85eeaafe008e11aa932a77c5c54c3297cfedd449765bc8bb53fa2aff04d38f" Nov 03 15:30:47.221570 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:47.221489 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb" Nov 03 15:30:48.231929 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:48.231892 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" event={"ID":"932d5d02-ff9f-40e8-bc1e-7f672c989547","Type":"ContainerStarted","Data":"8a44d028a5bcfc4325d17315125af479793bd364ad0bd96c03be29ff9a1b0934"} Nov 03 15:30:48.411410 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:48.411352 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" podStartSLOduration=2.411337304 podStartE2EDuration="2.411337304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:48.407592212 +0000 UTC m=+1358.250713745" watchObservedRunningTime="2026-11-03 15:30:48.411337304 +0000 UTC m=+1358.254458845" Nov 03 15:30:48.746962 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:48.746929 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:48.748132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:48.748107 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 15:30:48.748240 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:48.748154 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" podUID="932d5d02-ff9f-40e8-bc1e-7f672c989547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 15:30:49.747187 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:49.747147 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 15:30:49.747660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:49.747209 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" podUID="932d5d02-ff9f-40e8-bc1e-7f672c989547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 15:30:49.793376 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:49.793274 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:50.006659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:50.006565 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:50.202906 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:50.202875 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:50.747698 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:50.747660 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 15:30:50.748092 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:50.747717 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" podUID="932d5d02-ff9f-40e8-bc1e-7f672c989547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 15:30:51.420858 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:51.420817 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:51.627292 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:51.627260 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:51.747114 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:51.747029 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 15:30:51.747114 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:51.747083 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" podUID="932d5d02-ff9f-40e8-bc1e-7f672c989547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 15:30:52.746899 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:52.746862 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 15:30:52.747341 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:52.746920 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" podUID="932d5d02-ff9f-40e8-bc1e-7f672c989547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 15:30:53.502073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.502039 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv"] Nov 03 15:30:53.502665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.502646 4557 cpu_manager.go:414] "containerMap: removing container" podUID="4f472ac9-570a-4124-ba65-e9f81b49342d" containerName="revoke-keys" Nov 03 15:30:53.502665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.502665 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f472ac9-570a-4124-ba65-e9f81b49342d" containerName="revoke-keys" Nov 03 15:30:53.502837 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.502677 4557 cpu_manager.go:414] "containerMap: removing container" podUID="442544d0-79c6-486e-a3bd-163752dc0112" containerName="revoke-keys" Nov 03 15:30:53.502837 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.502684 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="442544d0-79c6-486e-a3bd-163752dc0112" containerName="revoke-keys" Nov 03 15:30:53.502937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.502876 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f472ac9-570a-4124-ba65-e9f81b49342d" containerName="revoke-keys" Nov 03 15:30:53.502937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.502888 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="442544d0-79c6-486e-a3bd-163752dc0112" containerName="revoke-keys" Nov 03 15:30:53.507887 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.507868 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv" Nov 03 15:30:53.610736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.610703 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:30:53.610955 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.610722 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:30:53.615202 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.615178 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv"] Nov 03 15:30:53.674522 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.674482 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e488d5c5-a263-4aab-9bde-705df88b1bd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea6ae55-rczwv\" (UID: \"e488d5c5-a263-4aab-9bde-705df88b1bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv" Nov 03 15:30:53.674883 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.674852 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2x72\" (UniqueName: \"kubernetes.io/projected/e488d5c5-a263-4aab-9bde-705df88b1bd8-kube-api-access-c2x72\") pod \"maas-api-revoke-sub-fea6ae55-rczwv\" (UID: \"e488d5c5-a263-4aab-9bde-705df88b1bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv" Nov 03 15:30:53.674976 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.674916 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e488d5c5-a263-4aab-9bde-705df88b1bd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea6ae55-rczwv\" (UID: \"e488d5c5-a263-4aab-9bde-705df88b1bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv" Nov 03 15:30:53.747084 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.747053 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 15:30:53.747448 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.747104 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" podUID="932d5d02-ff9f-40e8-bc1e-7f672c989547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 15:30:53.776684 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.776618 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e488d5c5-a263-4aab-9bde-705df88b1bd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea6ae55-rczwv\" (UID: \"e488d5c5-a263-4aab-9bde-705df88b1bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv" Nov 03 15:30:53.776812 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.776692 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2x72\" (UniqueName: \"kubernetes.io/projected/e488d5c5-a263-4aab-9bde-705df88b1bd8-kube-api-access-c2x72\") pod \"maas-api-revoke-sub-fea6ae55-rczwv\" (UID: \"e488d5c5-a263-4aab-9bde-705df88b1bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv" Nov 03 15:30:53.776812 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.776732 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e488d5c5-a263-4aab-9bde-705df88b1bd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea6ae55-rczwv\" (UID: \"e488d5c5-a263-4aab-9bde-705df88b1bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv" Nov 03 15:30:53.777292 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.777270 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e488d5c5-a263-4aab-9bde-705df88b1bd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea6ae55-rczwv\" (UID: \"e488d5c5-a263-4aab-9bde-705df88b1bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv" Nov 03 15:30:53.779247 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.779221 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e488d5c5-a263-4aab-9bde-705df88b1bd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea6ae55-rczwv\" (UID: \"e488d5c5-a263-4aab-9bde-705df88b1bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv" Nov 03 15:30:53.840202 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.840165 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2x72\" (UniqueName: \"kubernetes.io/projected/e488d5c5-a263-4aab-9bde-705df88b1bd8-kube-api-access-c2x72\") pod \"maas-api-revoke-sub-fea6ae55-rczwv\" (UID: \"e488d5c5-a263-4aab-9bde-705df88b1bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv" Nov 03 15:30:53.882363 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.882311 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r"] Nov 03 15:30:53.886703 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.886680 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r" Nov 03 15:30:53.946025 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.945993 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r"] Nov 03 15:30:53.979837 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.979790 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2bfb1a1-688c-4056-9195-6f08a265e92c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f2d8de3-x288r\" (UID: \"d2bfb1a1-688c-4056-9195-6f08a265e92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r" Nov 03 15:30:53.979989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.979970 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2bfb1a1-688c-4056-9195-6f08a265e92c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f2d8de3-x288r\" (UID: \"d2bfb1a1-688c-4056-9195-6f08a265e92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r" Nov 03 15:30:53.980211 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:53.980192 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl8ss\" (UniqueName: \"kubernetes.io/projected/d2bfb1a1-688c-4056-9195-6f08a265e92c-kube-api-access-hl8ss\") pod \"maas-api-revoke-sub-5f2d8de3-x288r\" (UID: \"d2bfb1a1-688c-4056-9195-6f08a265e92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r" Nov 03 15:30:54.081756 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:54.081722 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2bfb1a1-688c-4056-9195-6f08a265e92c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f2d8de3-x288r\" (UID: \"d2bfb1a1-688c-4056-9195-6f08a265e92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r" Nov 03 15:30:54.081926 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:54.081812 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2bfb1a1-688c-4056-9195-6f08a265e92c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f2d8de3-x288r\" (UID: \"d2bfb1a1-688c-4056-9195-6f08a265e92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r" Nov 03 15:30:54.081973 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:54.081959 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl8ss\" (UniqueName: \"kubernetes.io/projected/d2bfb1a1-688c-4056-9195-6f08a265e92c-kube-api-access-hl8ss\") pod \"maas-api-revoke-sub-5f2d8de3-x288r\" (UID: \"d2bfb1a1-688c-4056-9195-6f08a265e92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r" Nov 03 15:30:54.082549 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:54.082522 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2bfb1a1-688c-4056-9195-6f08a265e92c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f2d8de3-x288r\" (UID: \"d2bfb1a1-688c-4056-9195-6f08a265e92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r" Nov 03 15:30:54.084046 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:54.084026 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2bfb1a1-688c-4056-9195-6f08a265e92c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f2d8de3-x288r\" (UID: \"d2bfb1a1-688c-4056-9195-6f08a265e92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r" Nov 03 15:30:54.121334 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:54.121304 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv" Nov 03 15:30:54.144368 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:54.144343 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl8ss\" (UniqueName: \"kubernetes.io/projected/d2bfb1a1-688c-4056-9195-6f08a265e92c-kube-api-access-hl8ss\") pod \"maas-api-revoke-sub-5f2d8de3-x288r\" (UID: \"d2bfb1a1-688c-4056-9195-6f08a265e92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r" Nov 03 15:30:54.204030 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:54.203997 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r" Nov 03 15:30:54.506289 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:54.506249 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv"] Nov 03 15:30:54.508077 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:30:54.508037 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode488d5c5_a263_4aab_9bde_705df88b1bd8.slice/crio-9b32cd55fb8d93679740af4d56b12f060dca00f1481bbaabe679941c0a01239e WatchSource:0}: Error finding container 9b32cd55fb8d93679740af4d56b12f060dca00f1481bbaabe679941c0a01239e: Status 404 returned error can't find the container with id 9b32cd55fb8d93679740af4d56b12f060dca00f1481bbaabe679941c0a01239e Nov 03 15:30:54.630655 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:30:54.630620 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2bfb1a1_688c_4056_9195_6f08a265e92c.slice/crio-c89094b2ceda96fd09936316263de60110958d23936d1268c62bd0148cd810a2 WatchSource:0}: Error finding container c89094b2ceda96fd09936316263de60110958d23936d1268c62bd0148cd810a2: Status 404 returned error can't find the container with id c89094b2ceda96fd09936316263de60110958d23936d1268c62bd0148cd810a2 Nov 03 15:30:54.633242 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:54.633185 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r"] Nov 03 15:30:54.751003 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:54.750967 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:54.751506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:54.751481 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:54.752115 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:54.752093 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:30:55.113278 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:55.113251 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:55.296892 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:55.296855 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv" event={"ID":"e488d5c5-a263-4aab-9bde-705df88b1bd8","Type":"ContainerStarted","Data":"9d3aa5e271255dd09ca7098045f72270fc4c8cfb3fb1c0b12cf5418a9b3f1b23"} Nov 03 15:30:55.297067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:55.296917 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv" event={"ID":"e488d5c5-a263-4aab-9bde-705df88b1bd8","Type":"ContainerStarted","Data":"9b32cd55fb8d93679740af4d56b12f060dca00f1481bbaabe679941c0a01239e"} Nov 03 15:30:55.298713 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:55.298682 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r" event={"ID":"d2bfb1a1-688c-4056-9195-6f08a265e92c","Type":"ContainerStarted","Data":"1d2bd20e15567a5820a7c7a60d4626d82d648ca995d947332bc2f42ffcef3ddb"} Nov 03 15:30:55.298855 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:55.298734 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r" event={"ID":"d2bfb1a1-688c-4056-9195-6f08a265e92c","Type":"ContainerStarted","Data":"c89094b2ceda96fd09936316263de60110958d23936d1268c62bd0148cd810a2"} Nov 03 15:30:55.413015 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:55.412970 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv" podStartSLOduration=2.412955341 podStartE2EDuration="2.412955341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:55.412472976 +0000 UTC m=+1365.255594519" watchObservedRunningTime="2026-11-03 15:30:55.412955341 +0000 UTC m=+1365.256076882" Nov 03 15:30:55.536796 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:55.536744 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r" podStartSLOduration=2.536728841 podStartE2EDuration="2.536728841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:55.533691581 +0000 UTC m=+1365.376813122" watchObservedRunningTime="2026-11-03 15:30:55.536728841 +0000 UTC m=+1365.379850383" Nov 03 15:30:56.308023 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:56.307980 4557 generic.go:356] "Generic (PLEG): container finished" podID="e488d5c5-a263-4aab-9bde-705df88b1bd8" containerID="9d3aa5e271255dd09ca7098045f72270fc4c8cfb3fb1c0b12cf5418a9b3f1b23" exitCode=0 Nov 03 15:30:56.308445 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:56.308166 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv" event={"ID":"e488d5c5-a263-4aab-9bde-705df88b1bd8","Type":"ContainerDied","Data":"9d3aa5e271255dd09ca7098045f72270fc4c8cfb3fb1c0b12cf5418a9b3f1b23"} Nov 03 15:30:56.310255 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:56.310228 4557 generic.go:356] "Generic (PLEG): container finished" podID="d2bfb1a1-688c-4056-9195-6f08a265e92c" containerID="1d2bd20e15567a5820a7c7a60d4626d82d648ca995d947332bc2f42ffcef3ddb" exitCode=0 Nov 03 15:30:56.310372 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:56.310302 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r" event={"ID":"d2bfb1a1-688c-4056-9195-6f08a265e92c","Type":"ContainerDied","Data":"1d2bd20e15567a5820a7c7a60d4626d82d648ca995d947332bc2f42ffcef3ddb"} Nov 03 15:30:57.129211 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.129176 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6"] Nov 03 15:30:57.133244 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.133228 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6" Nov 03 15:30:57.192738 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.192706 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6"] Nov 03 15:30:57.328894 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.328860 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81c3afc1-1565-4482-8be4-dc95791b8426-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dbb6da8-2kvg6\" (UID: \"81c3afc1-1565-4482-8be4-dc95791b8426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6" Nov 03 15:30:57.329290 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.328984 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81c3afc1-1565-4482-8be4-dc95791b8426-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dbb6da8-2kvg6\" (UID: \"81c3afc1-1565-4482-8be4-dc95791b8426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6" Nov 03 15:30:57.329290 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.329143 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pdsl\" (UniqueName: \"kubernetes.io/projected/81c3afc1-1565-4482-8be4-dc95791b8426-kube-api-access-7pdsl\") pod \"maas-api-revoke-sub-7dbb6da8-2kvg6\" (UID: \"81c3afc1-1565-4482-8be4-dc95791b8426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6" Nov 03 15:30:57.431920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.430750 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81c3afc1-1565-4482-8be4-dc95791b8426-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dbb6da8-2kvg6\" (UID: \"81c3afc1-1565-4482-8be4-dc95791b8426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6" Nov 03 15:30:57.431920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.430861 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81c3afc1-1565-4482-8be4-dc95791b8426-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dbb6da8-2kvg6\" (UID: \"81c3afc1-1565-4482-8be4-dc95791b8426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6" Nov 03 15:30:57.431920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.430992 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pdsl\" (UniqueName: \"kubernetes.io/projected/81c3afc1-1565-4482-8be4-dc95791b8426-kube-api-access-7pdsl\") pod \"maas-api-revoke-sub-7dbb6da8-2kvg6\" (UID: \"81c3afc1-1565-4482-8be4-dc95791b8426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6" Nov 03 15:30:57.432476 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.432448 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81c3afc1-1565-4482-8be4-dc95791b8426-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dbb6da8-2kvg6\" (UID: \"81c3afc1-1565-4482-8be4-dc95791b8426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6" Nov 03 15:30:57.433817 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.433794 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81c3afc1-1565-4482-8be4-dc95791b8426-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dbb6da8-2kvg6\" (UID: \"81c3afc1-1565-4482-8be4-dc95791b8426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6" Nov 03 15:30:57.489640 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.489611 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r" Nov 03 15:30:57.494403 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.494378 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pdsl\" (UniqueName: \"kubernetes.io/projected/81c3afc1-1565-4482-8be4-dc95791b8426-kube-api-access-7pdsl\") pod \"maas-api-revoke-sub-7dbb6da8-2kvg6\" (UID: \"81c3afc1-1565-4482-8be4-dc95791b8426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6" Nov 03 15:30:57.515612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.515576 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv" Nov 03 15:30:57.632983 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.632951 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e488d5c5-a263-4aab-9bde-705df88b1bd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e488d5c5-a263-4aab-9bde-705df88b1bd8-maas-api-xks-ca\") pod \"e488d5c5-a263-4aab-9bde-705df88b1bd8\" (UID: \"e488d5c5-a263-4aab-9bde-705df88b1bd8\") " Nov 03 15:30:57.633150 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.633076 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2bfb1a1-688c-4056-9195-6f08a265e92c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2bfb1a1-688c-4056-9195-6f08a265e92c-maas-api-xks-ca\") pod \"d2bfb1a1-688c-4056-9195-6f08a265e92c\" (UID: \"d2bfb1a1-688c-4056-9195-6f08a265e92c\") " Nov 03 15:30:57.633150 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.633138 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e488d5c5-a263-4aab-9bde-705df88b1bd8-kube-api-access-c2x72\" (UniqueName: \"kubernetes.io/projected/e488d5c5-a263-4aab-9bde-705df88b1bd8-kube-api-access-c2x72\") pod \"e488d5c5-a263-4aab-9bde-705df88b1bd8\" (UID: \"e488d5c5-a263-4aab-9bde-705df88b1bd8\") " Nov 03 15:30:57.633358 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.633338 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2bfb1a1-688c-4056-9195-6f08a265e92c-kube-api-access-hl8ss\" (UniqueName: \"kubernetes.io/projected/d2bfb1a1-688c-4056-9195-6f08a265e92c-kube-api-access-hl8ss\") pod \"d2bfb1a1-688c-4056-9195-6f08a265e92c\" (UID: \"d2bfb1a1-688c-4056-9195-6f08a265e92c\") " Nov 03 15:30:57.633440 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.633383 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e488d5c5-a263-4aab-9bde-705df88b1bd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e488d5c5-a263-4aab-9bde-705df88b1bd8-maas-api-openshift-ca\") pod \"e488d5c5-a263-4aab-9bde-705df88b1bd8\" (UID: \"e488d5c5-a263-4aab-9bde-705df88b1bd8\") " Nov 03 15:30:57.633494 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.633455 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2bfb1a1-688c-4056-9195-6f08a265e92c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2bfb1a1-688c-4056-9195-6f08a265e92c-maas-api-openshift-ca\") pod \"d2bfb1a1-688c-4056-9195-6f08a265e92c\" (UID: \"d2bfb1a1-688c-4056-9195-6f08a265e92c\") " Nov 03 15:30:57.634009 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.633940 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e488d5c5-a263-4aab-9bde-705df88b1bd8-maas-api-openshift-ca" pod "e488d5c5-a263-4aab-9bde-705df88b1bd8" (UID: "e488d5c5-a263-4aab-9bde-705df88b1bd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:57.634009 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.633949 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2bfb1a1-688c-4056-9195-6f08a265e92c-maas-api-openshift-ca" pod "d2bfb1a1-688c-4056-9195-6f08a265e92c" (UID: "d2bfb1a1-688c-4056-9195-6f08a265e92c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:30:57.634184 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.634084 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e488d5c5-a263-4aab-9bde-705df88b1bd8-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:57.634184 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.634101 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2bfb1a1-688c-4056-9195-6f08a265e92c-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:57.635260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.635234 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e488d5c5-a263-4aab-9bde-705df88b1bd8-maas-api-xks-ca" pod "e488d5c5-a263-4aab-9bde-705df88b1bd8" (UID: "e488d5c5-a263-4aab-9bde-705df88b1bd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:30:57.635665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.635640 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2bfb1a1-688c-4056-9195-6f08a265e92c-maas-api-xks-ca" pod "d2bfb1a1-688c-4056-9195-6f08a265e92c" (UID: "d2bfb1a1-688c-4056-9195-6f08a265e92c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:30:57.635821 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.635802 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e488d5c5-a263-4aab-9bde-705df88b1bd8-kube-api-access-c2x72" pod "e488d5c5-a263-4aab-9bde-705df88b1bd8" (UID: "e488d5c5-a263-4aab-9bde-705df88b1bd8"). InnerVolumeSpecName "kube-api-access-c2x72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:57.635872 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.635830 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2bfb1a1-688c-4056-9195-6f08a265e92c-kube-api-access-hl8ss" pod "d2bfb1a1-688c-4056-9195-6f08a265e92c" (UID: "d2bfb1a1-688c-4056-9195-6f08a265e92c"). InnerVolumeSpecName "kube-api-access-hl8ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:30:57.736116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.736039 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e488d5c5-a263-4aab-9bde-705df88b1bd8-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:57.736116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.736065 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2bfb1a1-688c-4056-9195-6f08a265e92c-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:57.736116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.736075 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2x72\" (UniqueName: \"kubernetes.io/projected/e488d5c5-a263-4aab-9bde-705df88b1bd8-kube-api-access-c2x72\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:57.736116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.736084 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl8ss\" (UniqueName: \"kubernetes.io/projected/d2bfb1a1-688c-4056-9195-6f08a265e92c-kube-api-access-hl8ss\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:30:57.746497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:57.746465 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6" Nov 03 15:30:58.131942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:58.131914 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6"] Nov 03 15:30:58.311914 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:58.311879 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:30:58.326722 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:58.326701 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r" Nov 03 15:30:58.326825 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:58.326737 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r" event={"ID":"d2bfb1a1-688c-4056-9195-6f08a265e92c","Type":"ContainerDied","Data":"c89094b2ceda96fd09936316263de60110958d23936d1268c62bd0148cd810a2"} Nov 03 15:30:58.326825 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:58.326784 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c89094b2ceda96fd09936316263de60110958d23936d1268c62bd0148cd810a2" Nov 03 15:30:58.328230 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:58.328204 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6" event={"ID":"81c3afc1-1565-4482-8be4-dc95791b8426","Type":"ContainerStarted","Data":"84415d606582a3a8e1119e8d98855420c784e882a2b87690539744fc6a0e104c"} Nov 03 15:30:58.329700 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:58.329680 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv" event={"ID":"e488d5c5-a263-4aab-9bde-705df88b1bd8","Type":"ContainerDied","Data":"9b32cd55fb8d93679740af4d56b12f060dca00f1481bbaabe679941c0a01239e"} Nov 03 15:30:58.329700 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:58.329694 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv" Nov 03 15:30:58.330028 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:58.329721 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b32cd55fb8d93679740af4d56b12f060dca00f1481bbaabe679941c0a01239e" Nov 03 15:30:59.338772 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:59.338741 4557 generic.go:356] "Generic (PLEG): container finished" podID="81c3afc1-1565-4482-8be4-dc95791b8426" containerID="9f623813d98f304cdcf4fb1cca855cbe2eb4d48e28c1d01044c870c7529055c4" exitCode=0 Nov 03 15:30:59.339138 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:30:59.338780 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6" event={"ID":"81c3afc1-1565-4482-8be4-dc95791b8426","Type":"ContainerDied","Data":"9f623813d98f304cdcf4fb1cca855cbe2eb4d48e28c1d01044c870c7529055c4"} Nov 03 15:31:00.015353 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:00.015319 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:00.510199 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:00.510174 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6" Nov 03 15:31:00.571167 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:00.571140 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81c3afc1-1565-4482-8be4-dc95791b8426-kube-api-access-7pdsl\" (UniqueName: \"kubernetes.io/projected/81c3afc1-1565-4482-8be4-dc95791b8426-kube-api-access-7pdsl\") pod \"81c3afc1-1565-4482-8be4-dc95791b8426\" (UID: \"81c3afc1-1565-4482-8be4-dc95791b8426\") " Nov 03 15:31:00.571287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:00.571267 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81c3afc1-1565-4482-8be4-dc95791b8426-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81c3afc1-1565-4482-8be4-dc95791b8426-maas-api-xks-ca\") pod \"81c3afc1-1565-4482-8be4-dc95791b8426\" (UID: \"81c3afc1-1565-4482-8be4-dc95791b8426\") " Nov 03 15:31:00.571398 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:00.571386 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81c3afc1-1565-4482-8be4-dc95791b8426-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81c3afc1-1565-4482-8be4-dc95791b8426-maas-api-openshift-ca\") pod \"81c3afc1-1565-4482-8be4-dc95791b8426\" (UID: \"81c3afc1-1565-4482-8be4-dc95791b8426\") " Nov 03 15:31:00.571744 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:00.571718 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81c3afc1-1565-4482-8be4-dc95791b8426-maas-api-openshift-ca" pod "81c3afc1-1565-4482-8be4-dc95791b8426" (UID: "81c3afc1-1565-4482-8be4-dc95791b8426"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:31:00.571933 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:00.571917 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81c3afc1-1565-4482-8be4-dc95791b8426-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:31:00.573221 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:00.573194 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c3afc1-1565-4482-8be4-dc95791b8426-kube-api-access-7pdsl" pod "81c3afc1-1565-4482-8be4-dc95791b8426" (UID: "81c3afc1-1565-4482-8be4-dc95791b8426"). InnerVolumeSpecName "kube-api-access-7pdsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:31:00.573327 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:00.573224 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81c3afc1-1565-4482-8be4-dc95791b8426-maas-api-xks-ca" pod "81c3afc1-1565-4482-8be4-dc95791b8426" (UID: "81c3afc1-1565-4482-8be4-dc95791b8426"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:31:00.672911 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:00.672832 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pdsl\" (UniqueName: \"kubernetes.io/projected/81c3afc1-1565-4482-8be4-dc95791b8426-kube-api-access-7pdsl\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:31:00.672911 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:00.672861 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81c3afc1-1565-4482-8be4-dc95791b8426-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:31:01.354430 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.354390 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6" event={"ID":"81c3afc1-1565-4482-8be4-dc95791b8426","Type":"ContainerDied","Data":"84415d606582a3a8e1119e8d98855420c784e882a2b87690539744fc6a0e104c"} Nov 03 15:31:01.354620 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.354457 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84415d606582a3a8e1119e8d98855420c784e882a2b87690539744fc6a0e104c" Nov 03 15:31:01.354620 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.354400 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6" Nov 03 15:31:01.510812 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.510717 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z"] Nov 03 15:31:01.511475 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.511455 4557 cpu_manager.go:414] "containerMap: removing container" podUID="e488d5c5-a263-4aab-9bde-705df88b1bd8" containerName="revoke-keys" Nov 03 15:31:01.511572 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.511478 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e488d5c5-a263-4aab-9bde-705df88b1bd8" containerName="revoke-keys" Nov 03 15:31:01.511572 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.511502 4557 cpu_manager.go:414] "containerMap: removing container" podUID="d2bfb1a1-688c-4056-9195-6f08a265e92c" containerName="revoke-keys" Nov 03 15:31:01.511572 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.511509 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2bfb1a1-688c-4056-9195-6f08a265e92c" containerName="revoke-keys" Nov 03 15:31:01.511572 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.511525 4557 cpu_manager.go:414] "containerMap: removing container" podUID="81c3afc1-1565-4482-8be4-dc95791b8426" containerName="revoke-keys" Nov 03 15:31:01.511572 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.511533 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81c3afc1-1565-4482-8be4-dc95791b8426" containerName="revoke-keys" Nov 03 15:31:01.511865 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.511711 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2bfb1a1-688c-4056-9195-6f08a265e92c" containerName="revoke-keys" Nov 03 15:31:01.511865 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.511726 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="81c3afc1-1565-4482-8be4-dc95791b8426" containerName="revoke-keys" Nov 03 15:31:01.511865 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.511744 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="e488d5c5-a263-4aab-9bde-705df88b1bd8" containerName="revoke-keys" Nov 03 15:31:01.515158 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.515138 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.583751 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.583710 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.583937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.583798 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.583937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.583928 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.584055 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.583975 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.584101 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.584052 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.584143 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.584121 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.584255 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.584230 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.584335 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.584278 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.584380 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.584362 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgsll\" (UniqueName: \"kubernetes.io/projected/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-kube-api-access-hgsll\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.685272 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.685185 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.685272 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.685231 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.685464 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.685300 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.685464 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.685328 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.685464 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.685358 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.685464 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.685419 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.685680 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.685467 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.685680 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.685506 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.685680 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.685532 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgsll\" (UniqueName: \"kubernetes.io/projected/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-kube-api-access-hgsll\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.685833 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.685730 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.685833 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.685780 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.685945 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.685868 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.686121 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.686008 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.686340 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.686318 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.687813 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.687788 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.687952 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.687935 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.725657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.725628 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z"] Nov 03 15:31:01.760012 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.759980 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgsll\" (UniqueName: \"kubernetes.io/projected/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-kube-api-access-hgsll\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.760178 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.759981 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:01.828721 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:01.828690 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:02.266136 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:31:02.266106 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f3ff0c_bfe4_4b9d_b1f7_11907ca5cfe5.slice/crio-3c23fd7dba3dd9e00fab784506c4bda53e095fbb3213950a7e872ee6cef7d44c WatchSource:0}: Error finding container 3c23fd7dba3dd9e00fab784506c4bda53e095fbb3213950a7e872ee6cef7d44c: Status 404 returned error can't find the container with id 3c23fd7dba3dd9e00fab784506c4bda53e095fbb3213950a7e872ee6cef7d44c Nov 03 15:31:02.266955 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:02.266933 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z"] Nov 03 15:31:02.269150 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:02.269117 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:31:02.269252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:02.269198 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:31:02.269252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:02.269240 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:31:02.366953 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:02.366897 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" event={"ID":"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5","Type":"ContainerStarted","Data":"053f28f60a320524f23c3ffecf5c6e7e09508eb2c395d40f01b7e3996211eb93"} Nov 03 15:31:02.367177 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:02.366994 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" event={"ID":"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5","Type":"ContainerStarted","Data":"3c23fd7dba3dd9e00fab784506c4bda53e095fbb3213950a7e872ee6cef7d44c"} Nov 03 15:31:02.492967 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:02.492909 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" podStartSLOduration=1.492894913 podStartE2EDuration="1.492894913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:02.488101591 +0000 UTC m=+1372.331223149" watchObservedRunningTime="2026-11-03 15:31:02.492894913 +0000 UTC m=+1372.336016455" Nov 03 15:31:02.828858 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:02.828827 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:03.001891 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:03.001844 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:03.831255 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:03.831220 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 15:31:03.831753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:03.831271 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" podUID="24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 15:31:04.667760 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:04.667724 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:04.829024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:04.828996 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 15:31:04.829271 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:04.829248 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" podUID="24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 15:31:04.857763 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:04.857730 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:05.058433 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:05.058400 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:05.828969 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:05.828937 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 15:31:05.829202 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:05.829172 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" podUID="24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 15:31:06.829009 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:06.828974 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 15:31:06.829404 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:06.829028 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" podUID="24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 15:31:07.071754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:07.071725 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:07.843605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:07.829159 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:07.887960 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:07.887927 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 15:31:07.888226 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:07.888198 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" podUID="24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 15:31:07.964256 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:07.964213 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt"] Nov 03 15:31:07.975606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:07.969721 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt" Nov 03 15:31:07.998612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:07.997881 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt"] Nov 03 15:31:08.028923 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:08.028896 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:31:08.043105 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:08.042784 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:31:08.126385 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:08.126282 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d6f0c6b-9501-4e78-9080-79127bf92470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c35003dd-bkfkt\" (UID: \"1d6f0c6b-9501-4e78-9080-79127bf92470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt" Nov 03 15:31:08.126667 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:08.126643 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f0c6b-9501-4e78-9080-79127bf92470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c35003dd-bkfkt\" (UID: \"1d6f0c6b-9501-4e78-9080-79127bf92470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt" Nov 03 15:31:08.126836 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:08.126820 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-768mb\" (UniqueName: \"kubernetes.io/projected/1d6f0c6b-9501-4e78-9080-79127bf92470-kube-api-access-768mb\") pod \"maas-api-revoke-sub-c35003dd-bkfkt\" (UID: \"1d6f0c6b-9501-4e78-9080-79127bf92470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt" Nov 03 15:31:08.228738 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:08.228696 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d6f0c6b-9501-4e78-9080-79127bf92470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c35003dd-bkfkt\" (UID: \"1d6f0c6b-9501-4e78-9080-79127bf92470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt" Nov 03 15:31:08.229008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:08.228987 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f0c6b-9501-4e78-9080-79127bf92470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c35003dd-bkfkt\" (UID: \"1d6f0c6b-9501-4e78-9080-79127bf92470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt" Nov 03 15:31:08.229149 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:08.229135 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-768mb\" (UniqueName: \"kubernetes.io/projected/1d6f0c6b-9501-4e78-9080-79127bf92470-kube-api-access-768mb\") pod \"maas-api-revoke-sub-c35003dd-bkfkt\" (UID: \"1d6f0c6b-9501-4e78-9080-79127bf92470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt" Nov 03 15:31:08.229918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:08.229895 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f0c6b-9501-4e78-9080-79127bf92470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c35003dd-bkfkt\" (UID: \"1d6f0c6b-9501-4e78-9080-79127bf92470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt" Nov 03 15:31:08.233342 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:08.233309 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d6f0c6b-9501-4e78-9080-79127bf92470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c35003dd-bkfkt\" (UID: \"1d6f0c6b-9501-4e78-9080-79127bf92470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt" Nov 03 15:31:08.306539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:08.306504 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-768mb\" (UniqueName: \"kubernetes.io/projected/1d6f0c6b-9501-4e78-9080-79127bf92470-kube-api-access-768mb\") pod \"maas-api-revoke-sub-c35003dd-bkfkt\" (UID: \"1d6f0c6b-9501-4e78-9080-79127bf92470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt" Nov 03 15:31:08.328316 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:08.328287 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt" Nov 03 15:31:08.428076 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:08.427249 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:08.767736 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:31:08.767701 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6f0c6b_9501_4e78_9080_79127bf92470.slice/crio-2433575965eb681626cc68eaafd67e86bb0a43ff5d8dc103f84b94412683c08a WatchSource:0}: Error finding container 2433575965eb681626cc68eaafd67e86bb0a43ff5d8dc103f84b94412683c08a: Status 404 returned error can't find the container with id 2433575965eb681626cc68eaafd67e86bb0a43ff5d8dc103f84b94412683c08a Nov 03 15:31:08.770760 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:08.770734 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt"] Nov 03 15:31:08.834990 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:08.834949 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 15:31:08.835136 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:08.835012 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" podUID="24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 15:31:09.456735 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:09.456615 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt" event={"ID":"1d6f0c6b-9501-4e78-9080-79127bf92470","Type":"ContainerStarted","Data":"396592b567338ded5d33413a31b6c1efd84e5eab95f831d6d3b131895e202367"} Nov 03 15:31:09.456735 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:09.456679 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt" event={"ID":"1d6f0c6b-9501-4e78-9080-79127bf92470","Type":"ContainerStarted","Data":"2433575965eb681626cc68eaafd67e86bb0a43ff5d8dc103f84b94412683c08a"} Nov 03 15:31:09.576464 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:09.576408 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt" podStartSLOduration=2.576394746 podStartE2EDuration="2.576394746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:09.573174765 +0000 UTC m=+1379.416296308" watchObservedRunningTime="2026-11-03 15:31:09.576394746 +0000 UTC m=+1379.419516288" Nov 03 15:31:10.085378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:10.085326 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:10.210603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:10.204313 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:10.210603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:10.204788 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:10.210603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:10.205703 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:31:10.287528 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:10.287495 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:10.507662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:10.503978 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:10.507662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:10.504640 4557 generic.go:356] "Generic (PLEG): container finished" podID="1d6f0c6b-9501-4e78-9080-79127bf92470" containerID="396592b567338ded5d33413a31b6c1efd84e5eab95f831d6d3b131895e202367" exitCode=0 Nov 03 15:31:10.507662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:10.505944 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt" event={"ID":"1d6f0c6b-9501-4e78-9080-79127bf92470","Type":"ContainerDied","Data":"396592b567338ded5d33413a31b6c1efd84e5eab95f831d6d3b131895e202367"} Nov 03 15:31:10.580300 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:10.580239 4557 scope.go:122] "RemoveContainer" containerID="e371108c04645515d762de1c13024ce4bfe2c7770a112653f8eba6704a674a64" Nov 03 15:31:10.627398 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:10.627377 4557 scope.go:122] "RemoveContainer" containerID="4a6b66b7f98ae4ae99070e15bc5453a3a9f51c7bf892dbb75d4cbb1ee688ac13" Nov 03 15:31:10.639736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:10.639556 4557 scope.go:122] "RemoveContainer" containerID="160172312d88cdb8f7eeb2c5585b3f84bb82d3baae7380562c43e529d17d9b49" Nov 03 15:31:10.666253 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:10.665833 4557 scope.go:122] "RemoveContainer" containerID="1d7c61069d1575661344ae24f53c32c6c1cb7a8bc85b9ae848a5f2cd337b2e33" Nov 03 15:31:10.758747 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:10.753662 4557 scope.go:122] "RemoveContainer" containerID="e466ed77882d805189f176fe7e9b1fb8dc2fe987269f057e6c36bc86eeda8b2f" Nov 03 15:31:10.798653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:10.789172 4557 scope.go:122] "RemoveContainer" containerID="490a492ee33e82a91261cb56dd78e064467691262d0d1d0119881176ea715aeb" Nov 03 15:31:10.872805 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:10.872363 4557 scope.go:122] "RemoveContainer" containerID="406b7974f5f26cf5e2feef0ec572a56f705ad1ec4184dd22777a71ab4bac3fac" Nov 03 15:31:10.965792 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:10.965746 4557 scope.go:122] "RemoveContainer" containerID="cc84cebd8dfb4065b7f1c3d83ea97b5c3a34934a377706c11d2de2b94d33b493" Nov 03 15:31:11.038892 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:11.037760 4557 scope.go:122] "RemoveContainer" containerID="63bff8c04b1232c099b872b56c5d737ebc2729e132bc8cfd1082f24624fbec30" Nov 03 15:31:11.237617 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:11.225083 4557 scope.go:122] "RemoveContainer" containerID="a49e50c9ef4d8aa52aa032fc4857136b2c8df3187a4806c92a70f32b400d2eb5" Nov 03 15:31:11.509818 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:11.505782 4557 scope.go:122] "RemoveContainer" containerID="a3c0d827248f2677ad15f87d8f5b07e619eac01f906b9bb0bb8923f5f364c9ba" Nov 03 15:31:12.032370 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:12.032339 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:12.162493 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:12.161905 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 03 15:31:12.473162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:12.404791 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:12.498667 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:12.473442 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 15:31:12.625606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:12.622620 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 03 15:31:12.636475 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:12.627215 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:12.723119 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:12.723082 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:12.738021 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:12.737943 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt" Nov 03 15:31:12.840564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:12.832873 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 15:31:12.858315 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:12.857714 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:12.921796 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:12.920477 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d6f0c6b-9501-4e78-9080-79127bf92470-kube-api-access-768mb\" (UniqueName: \"kubernetes.io/projected/1d6f0c6b-9501-4e78-9080-79127bf92470-kube-api-access-768mb\") pod \"1d6f0c6b-9501-4e78-9080-79127bf92470\" (UID: \"1d6f0c6b-9501-4e78-9080-79127bf92470\") " Nov 03 15:31:12.921796 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:12.920599 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d6f0c6b-9501-4e78-9080-79127bf92470-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f0c6b-9501-4e78-9080-79127bf92470-maas-api-openshift-ca\") pod \"1d6f0c6b-9501-4e78-9080-79127bf92470\" (UID: \"1d6f0c6b-9501-4e78-9080-79127bf92470\") " Nov 03 15:31:12.921796 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:12.920747 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d6f0c6b-9501-4e78-9080-79127bf92470-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d6f0c6b-9501-4e78-9080-79127bf92470-maas-api-xks-ca\") pod \"1d6f0c6b-9501-4e78-9080-79127bf92470\" (UID: \"1d6f0c6b-9501-4e78-9080-79127bf92470\") " Nov 03 15:31:12.963605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:12.940911 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 15:31:12.963605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:12.941222 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:12.963605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:12.941386 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Nov 03 15:31:12.974904 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:12.974833 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d6f0c6b-9501-4e78-9080-79127bf92470-maas-api-openshift-ca" pod "1d6f0c6b-9501-4e78-9080-79127bf92470" (UID: "1d6f0c6b-9501-4e78-9080-79127bf92470"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:31:13.073171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.024035 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f0c6b-9501-4e78-9080-79127bf92470-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:31:13.084618 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.081735 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt" event={"ID":"1d6f0c6b-9501-4e78-9080-79127bf92470","Type":"ContainerDied","Data":"2433575965eb681626cc68eaafd67e86bb0a43ff5d8dc103f84b94412683c08a"} Nov 03 15:31:13.084618 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.081804 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2433575965eb681626cc68eaafd67e86bb0a43ff5d8dc103f84b94412683c08a" Nov 03 15:31:13.084618 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.081884 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt" Nov 03 15:31:13.174506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.164475 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 03 15:31:13.174506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.173128 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:31:13.187357 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.186921 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d6f0c6b-9501-4e78-9080-79127bf92470-maas-api-xks-ca" pod "1d6f0c6b-9501-4e78-9080-79127bf92470" (UID: "1d6f0c6b-9501-4e78-9080-79127bf92470"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:31:13.187357 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.187136 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6f0c6b-9501-4e78-9080-79127bf92470-kube-api-access-768mb" pod "1d6f0c6b-9501-4e78-9080-79127bf92470" (UID: "1d6f0c6b-9501-4e78-9080-79127bf92470"). InnerVolumeSpecName "kube-api-access-768mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:31:13.249540 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.237535 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:13.249540 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.241791 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-768mb\" (UniqueName: \"kubernetes.io/projected/1d6f0c6b-9501-4e78-9080-79127bf92470-kube-api-access-768mb\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:31:13.249540 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.241821 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d6f0c6b-9501-4e78-9080-79127bf92470-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:31:13.276653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.276528 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:13.450604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.450310 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded" Nov 03 15:31:13.806620 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.739305 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:13.806620 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.741204 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:31:13.982162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.957410 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:13.982162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.957821 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 15:31:13.982162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.978563 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:31:13.982162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:13.978651 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:31:14.215610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:14.212219 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 03 15:31:14.256657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:14.252882 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:14.256657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:14.253327 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:31:14.380602 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:14.365614 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 15:31:14.405186 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:14.389939 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:14.517302 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:14.517205 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:14.640606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:14.628678 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 03 15:31:14.640606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:14.634531 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:14.928606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:14.913120 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 15:31:14.944606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:14.942483 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:14.979606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:14.978688 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:15.004605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:14.979940 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:15.004605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:14.996163 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:31:15.004605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:14.920177 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Nov 03 15:31:15.224146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.224042 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:15.256042 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.254768 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:15.297637 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.288688 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:15.400305 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.400250 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:15.541659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.528803 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:15.541659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.530285 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:31:15.541659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.531904 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:31:15.541659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.531982 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" podUID="3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:15.567976 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.562967 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 15:31:15.567976 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.563520 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:31:15.591523 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.590702 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:15.645110 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.645062 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:15.646445 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.646421 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:31:15.656825 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.652454 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Nov 03 15:31:15.679544 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.676743 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:31:15.921449 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.921409 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:15.922274 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.922253 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:31:15.960670 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.960623 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:15.991754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.991046 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:15.997701 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.991101 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 15:31:15.997916 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.991575 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:31:15.998376 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.998311 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" podUID="e362b475-7544-4095-a465-24905a71c6c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:15.998652 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:15.998553 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:31:16.009137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:16.001020 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:16.080270 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:16.052374 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:16.146259 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:16.146228 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:31:16.248656 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:16.225759 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:16.250807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:16.250428 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": context deadline exceeded" Nov 03 15:31:16.277641 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:16.257301 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:16.415774 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:16.412914 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:16.415774 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:16.413047 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:16.576672 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:16.573178 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 15:31:16.649941 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:16.648214 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 03 15:31:16.665228 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:16.665188 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Nov 03 15:31:16.920747 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:16.920657 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh"] Nov 03 15:31:16.940234 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:16.940204 4557 cpu_manager.go:414] "containerMap: removing container" podUID="1d6f0c6b-9501-4e78-9080-79127bf92470" containerName="revoke-keys" Nov 03 15:31:16.940480 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:16.940461 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d6f0c6b-9501-4e78-9080-79127bf92470" containerName="revoke-keys" Nov 03 15:31:16.995629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:16.970016 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d6f0c6b-9501-4e78-9080-79127bf92470" containerName="revoke-keys" Nov 03 15:31:16.996186 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:16.957650 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:17.034730 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.019004 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:17.036564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.018934 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Nov 03 15:31:17.060782 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.047499 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh"] Nov 03 15:31:17.061865 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.061844 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:31:17.063594 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.058789 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 15:31:17.063594 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.060265 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb5a70-d410-4f8c-a6f8-167bc2cc4979-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh\" (UID: \"2ecb5a70-d410-4f8c-a6f8-167bc2cc4979\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh" Nov 03 15:31:17.063948 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.063883 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6w7f\" (UniqueName: \"kubernetes.io/projected/2ecb5a70-d410-4f8c-a6f8-167bc2cc4979-kube-api-access-l6w7f\") pod \"payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh\" (UID: \"2ecb5a70-d410-4f8c-a6f8-167bc2cc4979\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh" Nov 03 15:31:17.094031 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.047613 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh" Nov 03 15:31:17.094031 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.060346 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:17.189008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.188903 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-77d0e91e\"" Nov 03 15:31:17.197666 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.195602 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6w7f\" (UniqueName: \"kubernetes.io/projected/2ecb5a70-d410-4f8c-a6f8-167bc2cc4979-kube-api-access-l6w7f\") pod \"payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh\" (UID: \"2ecb5a70-d410-4f8c-a6f8-167bc2cc4979\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh" Nov 03 15:31:17.197666 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.195959 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb5a70-d410-4f8c-a6f8-167bc2cc4979-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh\" (UID: \"2ecb5a70-d410-4f8c-a6f8-167bc2cc4979\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh" Nov 03 15:31:17.198594 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.198516 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb5a70-d410-4f8c-a6f8-167bc2cc4979-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh\" (UID: \"2ecb5a70-d410-4f8c-a6f8-167bc2cc4979\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh" Nov 03 15:31:17.206621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.203665 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:17.206621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.204376 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk"] Nov 03 15:31:17.228647 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:31:17.219195 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:31:17.245929 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.245718 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:17.259629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.253666 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 03 15:31:17.307629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.307272 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 03 15:31:17.460762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.401340 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk"] Nov 03 15:31:17.460762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.401574 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:31:17.460762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.403305 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk" Nov 03 15:31:17.460762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.421006 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:17.460762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.434218 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6w7f\" (UniqueName: \"kubernetes.io/projected/2ecb5a70-d410-4f8c-a6f8-167bc2cc4979-kube-api-access-l6w7f\") pod \"payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh\" (UID: \"2ecb5a70-d410-4f8c-a6f8-167bc2cc4979\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh" Nov 03 15:31:17.460762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.455787 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:31:17.465681 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.465461 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:17.480102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.480076 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-77d0e91e-dockercfg-pc9x8\"" Nov 03 15:31:17.480361 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.480344 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:31:17.493928 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.493896 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lvjz\" (UniqueName: \"kubernetes.io/projected/2c732128-c679-47d6-97aa-c8fcb6daa895-kube-api-access-7lvjz\") pod \"payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk\" (UID: \"2c732128-c679-47d6-97aa-c8fcb6daa895\") " pod="openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk" Nov 03 15:31:17.494298 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.494273 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c732128-c679-47d6-97aa-c8fcb6daa895-plugins-config-volume\") pod \"payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk\" (UID: \"2c732128-c679-47d6-97aa-c8fcb6daa895\") " pod="openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk" Nov 03 15:31:17.495288 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.495269 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:31:17.496876 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.496858 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:31:17.497657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.497640 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh" Nov 03 15:31:17.521882 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.521854 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:31:17.560614 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.556649 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:31:17.603244 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.603214 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c732128-c679-47d6-97aa-c8fcb6daa895-plugins-config-volume\") pod \"payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk\" (UID: \"2c732128-c679-47d6-97aa-c8fcb6daa895\") " pod="openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk" Nov 03 15:31:17.605023 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.604984 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lvjz\" (UniqueName: \"kubernetes.io/projected/2c732128-c679-47d6-97aa-c8fcb6daa895-kube-api-access-7lvjz\") pod \"payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk\" (UID: \"2c732128-c679-47d6-97aa-c8fcb6daa895\") " pod="openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk" Nov 03 15:31:17.608725 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.604420 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c732128-c679-47d6-97aa-c8fcb6daa895-plugins-config-volume\") pod \"payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk\" (UID: \"2c732128-c679-47d6-97aa-c8fcb6daa895\") " pod="openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk" Nov 03 15:31:17.625743 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.625710 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:17.737827 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.672306 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Nov 03 15:31:17.794766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.776065 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:31:17.794766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.776158 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:31:17.929802 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.929722 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lvjz\" (UniqueName: \"kubernetes.io/projected/2c732128-c679-47d6-97aa-c8fcb6daa895-kube-api-access-7lvjz\") pod \"payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk\" (UID: \"2c732128-c679-47d6-97aa-c8fcb6daa895\") " pod="openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk" Nov 03 15:31:17.958658 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:17.958628 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:31:18.018901 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:18.018422 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk" Nov 03 15:31:18.312053 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:18.312019 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh"] Nov 03 15:31:18.342244 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:31:18.342208 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecb5a70_d410_4f8c_a6f8_167bc2cc4979.slice/crio-a0c5f758454b24ccbea28359ae04c87e33cdb2a92d2f13eae73d8c85cf05c18f WatchSource:0}: Error finding container a0c5f758454b24ccbea28359ae04c87e33cdb2a92d2f13eae73d8c85cf05c18f: Status 404 returned error can't find the container with id a0c5f758454b24ccbea28359ae04c87e33cdb2a92d2f13eae73d8c85cf05c18f Nov 03 15:31:18.506646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:18.506498 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk"] Nov 03 15:31:19.175487 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:19.175451 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk" event={"ID":"2c732128-c679-47d6-97aa-c8fcb6daa895","Type":"ContainerStarted","Data":"df440803d779b83dbe8253b85d4340662d2309511dfe7b0b39c95ff04275ac1c"} Nov 03 15:31:19.175988 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:19.175513 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk" event={"ID":"2c732128-c679-47d6-97aa-c8fcb6daa895","Type":"ContainerStarted","Data":"8c955a827120ca8c2343796a2f8976e83d8f2433f3c1309906617082fe050570"} Nov 03 15:31:19.175988 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:19.175562 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk" Nov 03 15:31:19.177207 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:19.177179 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh" event={"ID":"2ecb5a70-d410-4f8c-a6f8-167bc2cc4979","Type":"ContainerStarted","Data":"d97e3a87702b4d876e308d2a609c6bbdc580e854265427106bbc4d21f1cdb423"} Nov 03 15:31:19.177316 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:19.177223 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh" event={"ID":"2ecb5a70-d410-4f8c-a6f8-167bc2cc4979","Type":"ContainerStarted","Data":"a0c5f758454b24ccbea28359ae04c87e33cdb2a92d2f13eae73d8c85cf05c18f"} Nov 03 15:31:19.177316 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:19.177281 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh" Nov 03 15:31:19.297815 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:19.297764 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk" podStartSLOduration=3.29774963 podStartE2EDuration="3.29774963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:19.295444619 +0000 UTC m=+1389.138566161" watchObservedRunningTime="2026-11-03 15:31:19.29774963 +0000 UTC m=+1389.140871173" Nov 03 15:31:19.415020 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:19.414955 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh" podStartSLOduration=3.41493642 podStartE2EDuration="3.41493642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:19.410990533 +0000 UTC m=+1389.254112088" watchObservedRunningTime="2026-11-03 15:31:19.41493642 +0000 UTC m=+1389.258057958" Nov 03 15:31:21.452069 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:21.452034 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:21.661139 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:21.661104 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:22.025178 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:22.025146 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:22.217353 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:22.217312 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:22.436917 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:22.436879 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:27.397439 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:27.397309 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:28.704215 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:28.704184 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:28.894292 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:28.894262 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:30.186903 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:30.186871 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk" Nov 03 15:31:30.187276 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:30.186958 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh" Nov 03 15:31:31.843818 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:31.843786 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs"] Nov 03 15:31:31.847902 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:31.847881 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:31.916357 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:31.916329 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn"] Nov 03 15:31:31.920348 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:31.920332 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:31.946445 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:31.946413 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:31.946538 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:31.946462 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:31.946538 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:31.946515 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2ce135-8cc3-4b9a-9503-856817681c6b-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:31.946667 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:31.946595 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:31.946741 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:31.946723 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:31.946795 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:31.946779 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv4rt\" (UniqueName: \"kubernetes.io/projected/da2ce135-8cc3-4b9a-9503-856817681c6b-kube-api-access-cv4rt\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:31.946833 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:31.946822 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:31.946918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:31.946903 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:31.946984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:31.946970 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:31.960618 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:31.960567 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs"] Nov 03 15:31:32.010437 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.010401 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn"] Nov 03 15:31:32.053542 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.053494 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.053741 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.053566 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.053741 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.053623 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv4rt\" (UniqueName: \"kubernetes.io/projected/da2ce135-8cc3-4b9a-9503-856817681c6b-kube-api-access-cv4rt\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.053741 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.053669 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.053741 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.053708 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl4nd\" (UniqueName: \"kubernetes.io/projected/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-kube-api-access-cl4nd\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.054214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.053743 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.054214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.053815 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.054214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.053888 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.054214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.053921 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.054214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.053979 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.054214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.054028 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.054214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.054070 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2ce135-8cc3-4b9a-9503-856817681c6b-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.054214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.054125 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.054214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.054201 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.054622 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.054226 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.054622 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.054262 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.054622 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.054322 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.054622 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.054412 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.079346 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.079314 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.080975 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.080399 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.081142 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.080683 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.081225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.080916 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.081616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.081505 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2ce135-8cc3-4b9a-9503-856817681c6b-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.098920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.087049 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2"] Nov 03 15:31:32.098920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.092544 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:31:32.103284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.103227 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.143540 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.142236 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2"] Nov 03 15:31:32.143540 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.142758 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv4rt\" (UniqueName: \"kubernetes.io/projected/da2ce135-8cc3-4b9a-9503-856817681c6b-kube-api-access-cv4rt\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.143540 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.143125 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.143827 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.143689 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.145367 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.145337 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-cceaf27f-serving-cert\"" Nov 03 15:31:32.159633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.155969 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.159633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.156110 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.159633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.156137 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.159633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.156168 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.159633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.156211 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.159633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.156284 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.159633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.156316 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.159633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.156371 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl4nd\" (UniqueName: \"kubernetes.io/projected/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-kube-api-access-cl4nd\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.159633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.156403 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.159633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.159042 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.159633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.159533 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.160149 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.160128 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.162130 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.161696 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.162130 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.161889 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:32.162719 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.162699 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.163026 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.163005 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.168317 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.168259 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.223760 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.223725 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.228596 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.228554 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl4nd\" (UniqueName: \"kubernetes.io/projected/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-kube-api-access-cl4nd\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.233304 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.233281 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:32.258931 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.257613 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4qdx\" (UniqueName: \"kubernetes.io/projected/6735b185-18df-47d2-b6a3-c1ed601d009d-kube-api-access-c4qdx\") pod \"maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2\" (UID: \"6735b185-18df-47d2-b6a3-c1ed601d009d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:31:32.258931 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.257666 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6735b185-18df-47d2-b6a3-c1ed601d009d-metrics-certs\") pod \"maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2\" (UID: \"6735b185-18df-47d2-b6a3-c1ed601d009d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:31:32.258931 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.257728 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6735b185-18df-47d2-b6a3-c1ed601d009d-openshift-service-ca\") pod \"maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2\" (UID: \"6735b185-18df-47d2-b6a3-c1ed601d009d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:31:32.258931 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.257829 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6735b185-18df-47d2-b6a3-c1ed601d009d-maas-api-tls\") pod \"maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2\" (UID: \"6735b185-18df-47d2-b6a3-c1ed601d009d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:31:32.288248 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.288213 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:32.359137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.359046 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4qdx\" (UniqueName: \"kubernetes.io/projected/6735b185-18df-47d2-b6a3-c1ed601d009d-kube-api-access-c4qdx\") pod \"maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2\" (UID: \"6735b185-18df-47d2-b6a3-c1ed601d009d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:31:32.359137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.359107 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6735b185-18df-47d2-b6a3-c1ed601d009d-metrics-certs\") pod \"maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2\" (UID: \"6735b185-18df-47d2-b6a3-c1ed601d009d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:31:32.359327 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.359200 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6735b185-18df-47d2-b6a3-c1ed601d009d-openshift-service-ca\") pod \"maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2\" (UID: \"6735b185-18df-47d2-b6a3-c1ed601d009d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:31:32.359391 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.359330 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6735b185-18df-47d2-b6a3-c1ed601d009d-maas-api-tls\") pod \"maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2\" (UID: \"6735b185-18df-47d2-b6a3-c1ed601d009d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:31:32.359687 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:31:32.359666 4557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-serving-cert: secret "maas-api-e2e-ait-cceaf27f-serving-cert" not found Nov 03 15:31:32.359753 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:31:32.359743 4557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6735b185-18df-47d2-b6a3-c1ed601d009d-maas-api-tls podName:6735b185-18df-47d2-b6a3-c1ed601d009d nodeName:}" failed. No retries permitted until 2026-11-03 15:31:32.859721126 +0000 UTC m=+1402.702842665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6735b185-18df-47d2-b6a3-c1ed601d009d-maas-api-tls") pod "maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" (UID: "6735b185-18df-47d2-b6a3-c1ed601d009d") : secret "maas-api-e2e-ait-cceaf27f-serving-cert" not found Nov 03 15:31:32.361860 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.361833 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6735b185-18df-47d2-b6a3-c1ed601d009d-openshift-service-ca\") pod \"maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2\" (UID: \"6735b185-18df-47d2-b6a3-c1ed601d009d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:31:32.364923 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.363678 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6735b185-18df-47d2-b6a3-c1ed601d009d-metrics-certs\") pod \"maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2\" (UID: \"6735b185-18df-47d2-b6a3-c1ed601d009d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:31:32.426285 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.426254 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4qdx\" (UniqueName: \"kubernetes.io/projected/6735b185-18df-47d2-b6a3-c1ed601d009d-kube-api-access-c4qdx\") pod \"maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2\" (UID: \"6735b185-18df-47d2-b6a3-c1ed601d009d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:31:32.480202 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.480166 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:32.625040 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.624949 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs"] Nov 03 15:31:32.660535 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:31:32.660508 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2ce135_8cc3_4b9a_9503_856817681c6b.slice/crio-83231bd3a931c13ba34cc794ebafe2a3987d3d7f7978932d6ff953747ab1ab1e WatchSource:0}: Error finding container 83231bd3a931c13ba34cc794ebafe2a3987d3d7f7978932d6ff953747ab1ab1e: Status 404 returned error can't find the container with id 83231bd3a931c13ba34cc794ebafe2a3987d3d7f7978932d6ff953747ab1ab1e Nov 03 15:31:32.663332 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.663298 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:31:32.663420 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.663369 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:31:32.663420 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.663405 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:31:32.676771 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:31:32.676744 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56dd9a2a_8c82_42fa_aa29_8fd331733f3a.slice/crio-06e8211618559fae200afb5f62e331e6d14735852e53a6f769da8ee37beb7a0c WatchSource:0}: Error finding container 06e8211618559fae200afb5f62e331e6d14735852e53a6f769da8ee37beb7a0c: Status 404 returned error can't find the container with id 06e8211618559fae200afb5f62e331e6d14735852e53a6f769da8ee37beb7a0c Nov 03 15:31:32.679391 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.679356 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:31:32.679509 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.679426 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:31:32.679509 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.679470 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:31:32.680326 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.680236 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn"] Nov 03 15:31:32.684003 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.683972 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:32.870757 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.870719 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6735b185-18df-47d2-b6a3-c1ed601d009d-maas-api-tls\") pod \"maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2\" (UID: \"6735b185-18df-47d2-b6a3-c1ed601d009d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:31:32.873296 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.873271 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6735b185-18df-47d2-b6a3-c1ed601d009d-maas-api-tls\") pod \"maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2\" (UID: \"6735b185-18df-47d2-b6a3-c1ed601d009d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:31:32.952819 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:32.952728 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:33.050182 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:33.050147 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:31:33.352056 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:33.352017 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" event={"ID":"56dd9a2a-8c82-42fa-aa29-8fd331733f3a","Type":"ContainerStarted","Data":"0c13065666797a72e008ce9ccac5f6263c65459b0dcba7e1edea5a4ad215b547"} Nov 03 15:31:33.352256 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:33.352100 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" event={"ID":"56dd9a2a-8c82-42fa-aa29-8fd331733f3a","Type":"ContainerStarted","Data":"06e8211618559fae200afb5f62e331e6d14735852e53a6f769da8ee37beb7a0c"} Nov 03 15:31:33.353859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:33.353823 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" event={"ID":"da2ce135-8cc3-4b9a-9503-856817681c6b","Type":"ContainerStarted","Data":"b00e087aa449570081dacd7e14fc1013b1a01a14639c7ef82d098ce916646b33"} Nov 03 15:31:33.354004 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:33.353880 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" event={"ID":"da2ce135-8cc3-4b9a-9503-856817681c6b","Type":"ContainerStarted","Data":"83231bd3a931c13ba34cc794ebafe2a3987d3d7f7978932d6ff953747ab1ab1e"} Nov 03 15:31:33.499617 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:33.499352 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" podStartSLOduration=2.499332667 podStartE2EDuration="2.499332667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:33.475335937 +0000 UTC m=+1403.318457480" watchObservedRunningTime="2026-11-03 15:31:33.499332667 +0000 UTC m=+1403.342454209" Nov 03 15:31:33.532611 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:33.530405 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2"] Nov 03 15:31:33.542707 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:33.542675 4557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:31:33.628529 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:33.628393 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" podStartSLOduration=2.6283712120000002 podStartE2EDuration="2.628371212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:33.607878223 +0000 UTC m=+1403.450999766" watchObservedRunningTime="2026-11-03 15:31:33.628371212 +0000 UTC m=+1403.471492754" Nov 03 15:31:34.164053 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:34.163748 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:34.164796 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:34.164725 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 15:31:34.164878 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:34.164772 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" podUID="da2ce135-8cc3-4b9a-9503-856817681c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 15:31:34.234956 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:34.234766 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:34.237948 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:34.237916 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 15:31:34.238103 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:34.237973 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" podUID="56dd9a2a-8c82-42fa-aa29-8fd331733f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 15:31:34.372811 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:34.372774 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" event={"ID":"6735b185-18df-47d2-b6a3-c1ed601d009d","Type":"ContainerStarted","Data":"faa0a56cf39a246a8bb47b2164c3f6699e32e717e435b01bdcf52bec6762c5c0"} Nov 03 15:31:35.162942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:35.162910 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 15:31:35.163106 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:35.162958 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" podUID="da2ce135-8cc3-4b9a-9503-856817681c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 15:31:35.234488 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:35.233698 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 15:31:35.234488 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:35.233763 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" podUID="56dd9a2a-8c82-42fa-aa29-8fd331733f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 15:31:35.424659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:35.424165 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" event={"ID":"6735b185-18df-47d2-b6a3-c1ed601d009d","Type":"ContainerStarted","Data":"3c684f6bea412de8719b1335705d345f72cfcf103691f1af447d80db94942870"} Nov 03 15:31:35.437602 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:35.424887 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:31:35.563608 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:35.557468 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" podStartSLOduration=2.602813243 podStartE2EDuration="3.557448101s" totalImagesPullingTime="954.634858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:35.542081212 +0000 UTC m=+1405.385202755" watchObservedRunningTime="2026-11-03 15:31:35.557448101 +0000 UTC m=+1405.400569643" Nov 03 15:31:36.162722 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:36.162679 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 15:31:36.162891 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:36.162733 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" podUID="da2ce135-8cc3-4b9a-9503-856817681c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 15:31:36.234028 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:36.233996 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 15:31:36.234275 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:36.234252 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" podUID="56dd9a2a-8c82-42fa-aa29-8fd331733f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 15:31:37.162701 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:37.162668 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 15:31:37.163079 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:37.162734 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" podUID="da2ce135-8cc3-4b9a-9503-856817681c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 15:31:37.235970 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:37.235927 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 15:31:37.236141 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:37.235980 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" podUID="56dd9a2a-8c82-42fa-aa29-8fd331733f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 15:31:37.831924 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:37.831887 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:38.163500 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:38.163426 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 15:31:38.163500 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:38.163479 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" podUID="da2ce135-8cc3-4b9a-9503-856817681c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 15:31:38.234819 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:38.234789 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 15:31:38.235027 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:38.235007 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" podUID="56dd9a2a-8c82-42fa-aa29-8fd331733f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 15:31:39.162927 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:39.162896 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 15:31:39.163102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:39.162951 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" podUID="da2ce135-8cc3-4b9a-9503-856817681c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 15:31:39.233943 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:39.233912 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 15:31:39.234292 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:39.233962 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" podUID="56dd9a2a-8c82-42fa-aa29-8fd331733f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 15:31:40.178064 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:40.177040 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:40.178064 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:40.177468 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:40.178453 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:40.178402 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:31:40.240608 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:40.240562 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:40.240992 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:40.240859 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:40.241545 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:40.241526 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:31:42.457250 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:42.457222 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:31:42.845412 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:42.845379 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:44.862099 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:44.862064 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j"] Nov 03 15:31:44.866951 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:44.866932 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:44.964357 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:44.964316 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:44.964520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:44.964498 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:44.964597 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:44.964556 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:44.964690 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:44.964663 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:44.964816 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:44.964798 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:44.964884 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:44.964866 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:44.965001 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:44.964979 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:44.965067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:44.965054 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:44.965172 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:44.965156 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtzxr\" (UniqueName: \"kubernetes.io/projected/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-kube-api-access-rtzxr\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.066768 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.066738 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.066947 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.066781 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.066947 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.066799 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.066947 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.066857 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.066947 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.066934 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.067147 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.066978 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.067147 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.067045 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.067147 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.067132 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtzxr\" (UniqueName: \"kubernetes.io/projected/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-kube-api-access-rtzxr\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.067314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.067215 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.067314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.067267 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.067433 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.067337 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.067433 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.067354 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.067658 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.067638 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.067806 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.067787 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.069066 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.069046 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.069515 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.069493 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.080678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.080648 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j"] Nov 03 15:31:45.128410 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.128336 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.128410 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.128364 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtzxr\" (UniqueName: \"kubernetes.io/projected/6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e-kube-api-access-rtzxr\") pod \"maas-default-gateway-openshift-default-596dc9545f-5952j\" (UID: \"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.181263 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.181238 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:45.621101 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:31:45.621067 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7cfe96_7fdb_4ab2_b140_89d8acc9ab2e.slice/crio-efa960a6e4519cdf801cdba71875a7fb1a31df04d89ae3397854e5e994ed9b0a WatchSource:0}: Error finding container efa960a6e4519cdf801cdba71875a7fb1a31df04d89ae3397854e5e994ed9b0a: Status 404 returned error can't find the container with id efa960a6e4519cdf801cdba71875a7fb1a31df04d89ae3397854e5e994ed9b0a Nov 03 15:31:45.621829 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.621807 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j"] Nov 03 15:31:45.623593 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.623544 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:31:45.623710 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.623631 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:31:45.623710 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:45.623680 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:31:46.521501 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:46.521466 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" event={"ID":"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e","Type":"ContainerStarted","Data":"7ddde69970a5308015f51da65f56788fc55877f2f837d04b8703a42e1dcceacf"} Nov 03 15:31:46.521975 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:46.521523 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" event={"ID":"6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e","Type":"ContainerStarted","Data":"efa960a6e4519cdf801cdba71875a7fb1a31df04d89ae3397854e5e994ed9b0a"} Nov 03 15:31:46.535869 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:31:46.535829 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:31:46.756826 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:46.756772 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" podStartSLOduration=2.756735977 podStartE2EDuration="2.756735977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:46.702093446 +0000 UTC m=+1416.545214989" watchObservedRunningTime="2026-11-03 15:31:46.756735977 +0000 UTC m=+1416.599857518" Nov 03 15:31:47.201778 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:47.201726 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:47.203431 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:47.203399 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5952j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 15:31:47.203565 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:47.203459 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" podUID="6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 15:31:48.043520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:48.043469 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:48.184095 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:48.184059 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5952j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 15:31:48.184466 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:48.184429 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" podUID="6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 15:31:49.197604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:49.193963 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5952j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 15:31:49.197604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:49.194028 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" podUID="6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 15:31:50.181957 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:50.181918 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5952j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 15:31:50.182262 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:50.182230 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" podUID="6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 15:31:51.196706 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:51.196669 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5952j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 15:31:51.197281 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:51.197248 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" podUID="6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 15:31:52.182548 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:52.182510 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5952j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 15:31:52.182736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:52.182610 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" podUID="6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 15:31:53.033537 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:53.033472 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:53.181849 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:53.181812 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5952j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 15:31:53.182070 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:53.181873 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" podUID="6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 15:31:54.417448 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:54.417414 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:54.417957 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:54.417939 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:54.418677 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:54.418658 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" Nov 03 15:31:55.766112 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:55.766077 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:58.499757 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:58.499705 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 15:31:58.565794 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:58.565751 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:58.585719 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:58.585670 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 15:31:58.675132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:58.675097 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:31:58.689831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:58.686702 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:58.690150 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:58.690113 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:58.781832 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:58.781729 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:58.783130 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:58.782758 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 15:31:58.864662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:58.864437 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9x97t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:31:58.864662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:58.864541 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9x97t" podUID="8345cb5f-d59d-44a3-b858-5885db9bb03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:59.031639 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.014809 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:59.031639 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.018053 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:59.176606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.152077 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 15:31:59.219249 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.219169 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 03 15:31:59.219718 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.219682 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Nov 03 15:31:59.297539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.296713 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:59.501665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.501337 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:59.502143 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.501994 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:31:59.524606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.511709 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tr6nx" podUID="7f27f30d-07d4-4ed3-b1db-4645289d94fe" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:59.524606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.521676 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:31:59.524606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.521734 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" podUID="11e80ae6-ae0d-4857-b490-ed829f3b1412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:59.572604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.572209 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 15:31:59.572811 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.572617 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:31:59.624967 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.624034 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:59.624967 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.624536 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:31:59.784822 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.784716 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:59.818606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.787387 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:31:59.844606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.819297 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:32:00.085753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.085722 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:32:00.089602 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:31:59.885674 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 03 15:32:00.105723 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.002124 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:32:00.105956 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.105925 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" podUID="7ee31efd-a014-4cfa-8578-bed65d98bfea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:00.106073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.029482 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Nov 03 15:32:00.118639 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.029039 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 03 15:32:00.126605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.119154 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:32:00.131651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.126859 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:32:00.131894 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.131872 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:32:00.183710 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.183664 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:00.204860 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.188701 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:32:00.506675 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.447212 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 03 15:32:00.506675 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.449887 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Nov 03 15:32:00.506675 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.449981 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 03 15:32:00.506675 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.502203 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:00.542606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.517178 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:32:00.543000 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.542974 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:32:00.543194 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.543179 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:32:00.582864 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.582734 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:00.629795 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.629757 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:00.785489 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.785394 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 15:32:00.831677 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.828681 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:01.028607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.992219 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded" Nov 03 15:32:01.028607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:00.992778 4557 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded" Nov 03 15:32:01.113849 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:01.113780 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:01.140606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:01.114478 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Nov 03 15:32:01.167629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:01.141964 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:01.195628 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:01.188921 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:01.576083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:01.459983 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 03 15:32:01.576083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:01.460332 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:01.576083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:01.536132 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:01.576083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:01.544235 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:32:01.576083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:01.548845 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:01.638112 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:01.637264 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 15:32:01.715325 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:01.712956 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:01.857827 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:01.857728 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:02.001722 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:01.990908 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:32:02.001722 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:01.990987 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:32:02.146669 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:02.134129 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:02.170006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:02.137650 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:02.170006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:02.149856 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:02.170006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:02.151340 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:32:02.682734 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:02.682686 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:02.791166 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:02.791119 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:02.948700 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:02.944287 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 03 15:32:02.958063 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:02.948148 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:32:02.971176 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:02.971147 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:32:02.971325 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:02.971219 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:32:02.971325 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:02.971254 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:32:02.971450 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:02.971371 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:32:02.971450 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:02.971410 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:32:03.028607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:03.004831 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:32:03.053762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:03.053714 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:32:03.620102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:03.620068 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:04.309459 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:04.309424 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:08.911191 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:08.911155 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x"] Nov 03 15:32:08.916943 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:08.916920 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:32:08.969343 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:08.969296 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-metrics-certs\") pod \"maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x\" (UID: \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:32:08.969534 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:08.969434 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-maas-api-tls\") pod \"maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x\" (UID: \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:32:08.969633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:08.969610 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-openshift-service-ca\") pod \"maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x\" (UID: \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:32:08.969728 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:08.969710 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5dvm\" (UniqueName: \"kubernetes.io/projected/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-kube-api-access-j5dvm\") pod \"maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x\" (UID: \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:32:08.976834 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:08.976806 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x"] Nov 03 15:32:09.022934 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:09.022864 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-77d0e91e-serving-cert\"" Nov 03 15:32:09.071433 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:09.071393 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-openshift-service-ca\") pod \"maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x\" (UID: \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:32:09.071763 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:09.071737 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5dvm\" (UniqueName: \"kubernetes.io/projected/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-kube-api-access-j5dvm\") pod \"maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x\" (UID: \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:32:09.072650 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:09.072136 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-metrics-certs\") pod \"maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x\" (UID: \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:32:09.072910 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:09.072292 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-openshift-service-ca\") pod \"maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x\" (UID: \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:32:09.072994 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:09.072892 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-maas-api-tls\") pod \"maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x\" (UID: \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:32:09.072994 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:32:09.072968 4557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-serving-cert: secret "maas-api-e2e-ait-77d0e91e-serving-cert" not found Nov 03 15:32:09.073114 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:32:09.073051 4557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-maas-api-tls podName:79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32 nodeName:}" failed. No retries permitted until 2026-11-03 15:32:09.573026866 +0000 UTC m=+1439.416148403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-maas-api-tls") pod "maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" (UID: "79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32") : secret "maas-api-e2e-ait-77d0e91e-serving-cert" not found Nov 03 15:32:09.074926 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:09.074906 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-metrics-certs\") pod \"maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x\" (UID: \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:32:09.150070 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:09.150038 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5dvm\" (UniqueName: \"kubernetes.io/projected/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-kube-api-access-j5dvm\") pod \"maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x\" (UID: \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:32:09.177938 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:09.177833 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:09.393769 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:09.393739 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:09.415749 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:09.415715 4557 patch_prober.go:29] interesting pod/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:32:09.415953 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:09.415773 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" podUID="4b57651a-2c8b-4df4-84c2-4f5fba809417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:32:09.586057 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:09.586022 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-maas-api-tls\") pod \"maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x\" (UID: \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:32:09.588487 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:09.588457 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-maas-api-tls\") pod \"maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x\" (UID: \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:32:09.593104 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:09.593073 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:09.832297 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:09.832265 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:32:10.234713 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:32:10.234682 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ffc7d6_69ab_4c8f_9773_aa0a4daf7e32.slice/crio-5a69587e4630f945fcb9f4173470900552c0b4b7544ff8152cd60369316c7300 WatchSource:0}: Error finding container 5a69587e4630f945fcb9f4173470900552c0b4b7544ff8152cd60369316c7300: Status 404 returned error can't find the container with id 5a69587e4630f945fcb9f4173470900552c0b4b7544ff8152cd60369316c7300 Nov 03 15:32:10.237902 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:10.237879 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x"] Nov 03 15:32:11.117859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:11.117818 4557 generic.go:356] "Generic (PLEG): container finished" podID="4b57651a-2c8b-4df4-84c2-4f5fba809417" containerID="e6c9817163d5e9bd3b616de0cc4ce8359718ab7abd15f2e98c58b02312bc73cb" exitCode=0 Nov 03 15:32:11.118243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:11.118072 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" event={"ID":"4b57651a-2c8b-4df4-84c2-4f5fba809417","Type":"ContainerDied","Data":"e6c9817163d5e9bd3b616de0cc4ce8359718ab7abd15f2e98c58b02312bc73cb"} Nov 03 15:32:11.118609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:11.118569 4557 scope.go:122] "RemoveContainer" containerID="e6c9817163d5e9bd3b616de0cc4ce8359718ab7abd15f2e98c58b02312bc73cb" Nov 03 15:32:11.122436 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:11.121113 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" event={"ID":"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32","Type":"ContainerStarted","Data":"5a69587e4630f945fcb9f4173470900552c0b4b7544ff8152cd60369316c7300"} Nov 03 15:32:11.127528 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:11.127494 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:32:11.127691 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:11.127569 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:32:11.127691 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:11.127618 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:32:12.136093 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:12.136051 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" event={"ID":"4b57651a-2c8b-4df4-84c2-4f5fba809417","Type":"ContainerStarted","Data":"f1678df45b3e48ef7df977e943f5a7de56f3a1a07b4d3f144a7661066915863d"} Nov 03 15:32:12.137862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:12.137838 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" event={"ID":"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32","Type":"ContainerStarted","Data":"faadc153354e149f9ce269bc5660da0b5cc7486ccc5cf0e6308cb665bc2736d5"} Nov 03 15:32:12.137984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:12.137901 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:32:12.312836 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:12.312792 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:32:12.313631 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:12.313548 4557 patch_prober.go:29] interesting pod/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:32:12.313767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:12.313626 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" podUID="4b57651a-2c8b-4df4-84c2-4f5fba809417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 15:32:12.401752 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:12.400967 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" podStartSLOduration=3.506171511 podStartE2EDuration="4.400946351s" totalImagesPullingTime="894.77484ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:32:12.40002136 +0000 UTC m=+1442.243142902" watchObservedRunningTime="2026-11-03 15:32:12.400946351 +0000 UTC m=+1442.244067908" Nov 03 15:32:13.270259 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:13.270228 4557 patch_prober.go:29] interesting pod/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:32:13.270776 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:13.270284 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" podUID="4b57651a-2c8b-4df4-84c2-4f5fba809417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 15:32:14.270877 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:14.270844 4557 patch_prober.go:29] interesting pod/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:32:14.271353 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:14.270901 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" podUID="4b57651a-2c8b-4df4-84c2-4f5fba809417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 15:32:14.308113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:14.308079 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:15.020032 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:15.020002 4557 scope.go:122] "RemoveContainer" containerID="eb5545a6cedb5da19825d6ef1eb93f6bddec50cc13b13d6c4370dd302b867fe8" Nov 03 15:32:15.029981 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:15.029959 4557 scope.go:122] "RemoveContainer" containerID="4ecdb6d403f014ea9e8fec04883d5f50e7947643364bd54ded92c60567dc0788" Nov 03 15:32:15.038817 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:15.038800 4557 scope.go:122] "RemoveContainer" containerID="ee877b497291ac84b1ba86e77fe8d9781766a7f9fc0f4f41dd5b29b41ca97855" Nov 03 15:32:15.271055 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:15.270975 4557 patch_prober.go:29] interesting pod/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:32:15.271055 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:15.271033 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" podUID="4b57651a-2c8b-4df4-84c2-4f5fba809417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 15:32:16.270727 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:16.270691 4557 patch_prober.go:29] interesting pod/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:32:16.270932 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:16.270757 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" podUID="4b57651a-2c8b-4df4-84c2-4f5fba809417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 15:32:16.533477 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:32:16.533392 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:32:17.270452 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:17.270415 4557 patch_prober.go:29] interesting pod/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 15:32:17.270646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:17.270495 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" podUID="4b57651a-2c8b-4df4-84c2-4f5fba809417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 15:32:18.162947 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:18.162909 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:32:18.275273 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:18.275235 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:32:18.275640 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:18.275619 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:32:18.276328 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:18.276305 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:32:19.328071 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:19.328040 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:22.878866 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:22.878832 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn"] Nov 03 15:32:22.885025 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:22.885002 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn" Nov 03 15:32:22.942234 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:22.942207 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn"] Nov 03 15:32:22.985095 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:22.985047 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:32:22.985262 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:22.985099 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:32:23.017009 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:23.016970 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2fb33ba-ad6a-4bc3-8209-792eff19c786-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-47b989b3-5jttn\" (UID: \"e2fb33ba-ad6a-4bc3-8209-792eff19c786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn" Nov 03 15:32:23.017167 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:23.017122 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drxhb\" (UniqueName: \"kubernetes.io/projected/e2fb33ba-ad6a-4bc3-8209-792eff19c786-kube-api-access-drxhb\") pod \"maas-api-revoke-keys-e2e-ait-47b989b3-5jttn\" (UID: \"e2fb33ba-ad6a-4bc3-8209-792eff19c786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn" Nov 03 15:32:23.017263 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:23.017189 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2fb33ba-ad6a-4bc3-8209-792eff19c786-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-47b989b3-5jttn\" (UID: \"e2fb33ba-ad6a-4bc3-8209-792eff19c786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn" Nov 03 15:32:23.119129 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:23.119082 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2fb33ba-ad6a-4bc3-8209-792eff19c786-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-47b989b3-5jttn\" (UID: \"e2fb33ba-ad6a-4bc3-8209-792eff19c786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn" Nov 03 15:32:23.119313 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:23.119291 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxhb\" (UniqueName: \"kubernetes.io/projected/e2fb33ba-ad6a-4bc3-8209-792eff19c786-kube-api-access-drxhb\") pod \"maas-api-revoke-keys-e2e-ait-47b989b3-5jttn\" (UID: \"e2fb33ba-ad6a-4bc3-8209-792eff19c786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn" Nov 03 15:32:23.119403 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:23.119386 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2fb33ba-ad6a-4bc3-8209-792eff19c786-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-47b989b3-5jttn\" (UID: \"e2fb33ba-ad6a-4bc3-8209-792eff19c786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn" Nov 03 15:32:23.119957 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:23.119929 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2fb33ba-ad6a-4bc3-8209-792eff19c786-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-47b989b3-5jttn\" (UID: \"e2fb33ba-ad6a-4bc3-8209-792eff19c786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn" Nov 03 15:32:23.121752 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:23.121733 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2fb33ba-ad6a-4bc3-8209-792eff19c786-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-47b989b3-5jttn\" (UID: \"e2fb33ba-ad6a-4bc3-8209-792eff19c786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn" Nov 03 15:32:23.181159 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:23.181093 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drxhb\" (UniqueName: \"kubernetes.io/projected/e2fb33ba-ad6a-4bc3-8209-792eff19c786-kube-api-access-drxhb\") pod \"maas-api-revoke-keys-e2e-ait-47b989b3-5jttn\" (UID: \"e2fb33ba-ad6a-4bc3-8209-792eff19c786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn" Nov 03 15:32:23.199009 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:23.198985 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn" Nov 03 15:32:23.591777 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:32:23.591741 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2fb33ba_ad6a_4bc3_8209_792eff19c786.slice/crio-ba4c93500b8e8ba50aa5fba27e813f57127831048165bca6ff744acb5413e5ae WatchSource:0}: Error finding container ba4c93500b8e8ba50aa5fba27e813f57127831048165bca6ff744acb5413e5ae: Status 404 returned error can't find the container with id ba4c93500b8e8ba50aa5fba27e813f57127831048165bca6ff744acb5413e5ae Nov 03 15:32:23.593139 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:23.593114 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn"] Nov 03 15:32:24.239227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:24.239143 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn" event={"ID":"e2fb33ba-ad6a-4bc3-8209-792eff19c786","Type":"ContainerStarted","Data":"523594088ea5349b09daa86892727238d2b30212314a3779fc658c3663c753bf"} Nov 03 15:32:24.239227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:24.239200 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn" event={"ID":"e2fb33ba-ad6a-4bc3-8209-792eff19c786","Type":"ContainerStarted","Data":"ba4c93500b8e8ba50aa5fba27e813f57127831048165bca6ff744acb5413e5ae"} Nov 03 15:32:24.359675 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:24.359623 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn" podStartSLOduration=2.3596089 podStartE2EDuration="2.3596089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:32:24.355191486 +0000 UTC m=+1454.198313031" watchObservedRunningTime="2026-11-03 15:32:24.3596089 +0000 UTC m=+1454.202730452" Nov 03 15:32:24.588938 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:24.588906 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:25.249743 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:25.249702 4557 generic.go:356] "Generic (PLEG): container finished" podID="e2fb33ba-ad6a-4bc3-8209-792eff19c786" containerID="523594088ea5349b09daa86892727238d2b30212314a3779fc658c3663c753bf" exitCode=0 Nov 03 15:32:25.250117 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:25.249783 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn" event={"ID":"e2fb33ba-ad6a-4bc3-8209-792eff19c786","Type":"ContainerDied","Data":"523594088ea5349b09daa86892727238d2b30212314a3779fc658c3663c753bf"} Nov 03 15:32:26.437994 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:26.437964 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn" Nov 03 15:32:26.569648 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:26.569571 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2fb33ba-ad6a-4bc3-8209-792eff19c786-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2fb33ba-ad6a-4bc3-8209-792eff19c786-maas-api-openshift-ca\") pod \"e2fb33ba-ad6a-4bc3-8209-792eff19c786\" (UID: \"e2fb33ba-ad6a-4bc3-8209-792eff19c786\") " Nov 03 15:32:26.569820 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:26.569734 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2fb33ba-ad6a-4bc3-8209-792eff19c786-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2fb33ba-ad6a-4bc3-8209-792eff19c786-maas-api-xks-ca\") pod \"e2fb33ba-ad6a-4bc3-8209-792eff19c786\" (UID: \"e2fb33ba-ad6a-4bc3-8209-792eff19c786\") " Nov 03 15:32:26.569916 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:26.569892 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2fb33ba-ad6a-4bc3-8209-792eff19c786-maas-api-openshift-ca" pod "e2fb33ba-ad6a-4bc3-8209-792eff19c786" (UID: "e2fb33ba-ad6a-4bc3-8209-792eff19c786"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:32:26.569993 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:26.569919 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2fb33ba-ad6a-4bc3-8209-792eff19c786-kube-api-access-drxhb\" (UniqueName: \"kubernetes.io/projected/e2fb33ba-ad6a-4bc3-8209-792eff19c786-kube-api-access-drxhb\") pod \"e2fb33ba-ad6a-4bc3-8209-792eff19c786\" (UID: \"e2fb33ba-ad6a-4bc3-8209-792eff19c786\") " Nov 03 15:32:26.570566 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:26.570540 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2fb33ba-ad6a-4bc3-8209-792eff19c786-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:26.571915 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:26.571887 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2fb33ba-ad6a-4bc3-8209-792eff19c786-maas-api-xks-ca" pod "e2fb33ba-ad6a-4bc3-8209-792eff19c786" (UID: "e2fb33ba-ad6a-4bc3-8209-792eff19c786"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:32:26.572906 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:26.572883 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2fb33ba-ad6a-4bc3-8209-792eff19c786-kube-api-access-drxhb" pod "e2fb33ba-ad6a-4bc3-8209-792eff19c786" (UID: "e2fb33ba-ad6a-4bc3-8209-792eff19c786"). InnerVolumeSpecName "kube-api-access-drxhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:32:26.672336 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:26.672289 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drxhb\" (UniqueName: \"kubernetes.io/projected/e2fb33ba-ad6a-4bc3-8209-792eff19c786-kube-api-access-drxhb\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:26.672336 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:26.672331 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2fb33ba-ad6a-4bc3-8209-792eff19c786-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:27.266058 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.266032 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn" Nov 03 15:32:27.266262 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.266067 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn" event={"ID":"e2fb33ba-ad6a-4bc3-8209-792eff19c786","Type":"ContainerDied","Data":"ba4c93500b8e8ba50aa5fba27e813f57127831048165bca6ff744acb5413e5ae"} Nov 03 15:32:27.266262 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.266115 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba4c93500b8e8ba50aa5fba27e813f57127831048165bca6ff744acb5413e5ae" Nov 03 15:32:27.388702 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.388667 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh"] Nov 03 15:32:27.390204 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.390176 4557 cpu_manager.go:414] "containerMap: removing container" podUID="e2fb33ba-ad6a-4bc3-8209-792eff19c786" containerName="revoke-keys" Nov 03 15:32:27.390369 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.390353 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2fb33ba-ad6a-4bc3-8209-792eff19c786" containerName="revoke-keys" Nov 03 15:32:27.391716 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.391694 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2fb33ba-ad6a-4bc3-8209-792eff19c786" containerName="revoke-keys" Nov 03 15:32:27.397392 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.397374 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh" Nov 03 15:32:27.404247 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.404222 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh"] Nov 03 15:32:27.450801 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.450759 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:32:27.450801 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.450794 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:32:27.495683 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.495650 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgdf9\" (UniqueName: \"kubernetes.io/projected/23fb2af4-78c2-4361-8243-e98c35246703-kube-api-access-kgdf9\") pod \"maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh\" (UID: \"23fb2af4-78c2-4361-8243-e98c35246703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh" Nov 03 15:32:27.495865 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.495712 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23fb2af4-78c2-4361-8243-e98c35246703-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh\" (UID: \"23fb2af4-78c2-4361-8243-e98c35246703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh" Nov 03 15:32:27.496050 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.496027 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23fb2af4-78c2-4361-8243-e98c35246703-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh\" (UID: \"23fb2af4-78c2-4361-8243-e98c35246703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh" Nov 03 15:32:27.597838 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.597799 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23fb2af4-78c2-4361-8243-e98c35246703-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh\" (UID: \"23fb2af4-78c2-4361-8243-e98c35246703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh" Nov 03 15:32:27.598021 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.597888 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgdf9\" (UniqueName: \"kubernetes.io/projected/23fb2af4-78c2-4361-8243-e98c35246703-kube-api-access-kgdf9\") pod \"maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh\" (UID: \"23fb2af4-78c2-4361-8243-e98c35246703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh" Nov 03 15:32:27.598021 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.597943 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23fb2af4-78c2-4361-8243-e98c35246703-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh\" (UID: \"23fb2af4-78c2-4361-8243-e98c35246703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh" Nov 03 15:32:27.598504 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.598484 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23fb2af4-78c2-4361-8243-e98c35246703-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh\" (UID: \"23fb2af4-78c2-4361-8243-e98c35246703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh" Nov 03 15:32:27.600282 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.600250 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23fb2af4-78c2-4361-8243-e98c35246703-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh\" (UID: \"23fb2af4-78c2-4361-8243-e98c35246703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh" Nov 03 15:32:27.659873 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.659846 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgdf9\" (UniqueName: \"kubernetes.io/projected/23fb2af4-78c2-4361-8243-e98c35246703-kube-api-access-kgdf9\") pod \"maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh\" (UID: \"23fb2af4-78c2-4361-8243-e98c35246703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh" Nov 03 15:32:27.716244 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:27.716209 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh" Nov 03 15:32:28.094641 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:32:28.094613 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23fb2af4_78c2_4361_8243_e98c35246703.slice/crio-27e267ffb04409fb36f224c55f32f8531ecccb32a3683fa9efb6ea611c9576a4 WatchSource:0}: Error finding container 27e267ffb04409fb36f224c55f32f8531ecccb32a3683fa9efb6ea611c9576a4: Status 404 returned error can't find the container with id 27e267ffb04409fb36f224c55f32f8531ecccb32a3683fa9efb6ea611c9576a4 Nov 03 15:32:28.095674 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:28.095648 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh"] Nov 03 15:32:28.274431 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:28.274393 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh" event={"ID":"23fb2af4-78c2-4361-8243-e98c35246703","Type":"ContainerStarted","Data":"27e267ffb04409fb36f224c55f32f8531ecccb32a3683fa9efb6ea611c9576a4"} Nov 03 15:32:28.468609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:28.468552 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:28.664520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:28.664449 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:28.898067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:28.898031 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:29.283008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:29.282928 4557 generic.go:356] "Generic (PLEG): container finished" podID="23fb2af4-78c2-4361-8243-e98c35246703" containerID="138d32feb911ebb6697ff62b15caa67006157d780b26727b01971faa9c49632e" exitCode=0 Nov 03 15:32:29.283154 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:29.283013 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh" event={"ID":"23fb2af4-78c2-4361-8243-e98c35246703","Type":"ContainerDied","Data":"138d32feb911ebb6697ff62b15caa67006157d780b26727b01971faa9c49632e"} Nov 03 15:32:29.811697 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:29.811659 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:30.464037 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:30.464009 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh" Nov 03 15:32:30.536871 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:30.536845 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23fb2af4-78c2-4361-8243-e98c35246703-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23fb2af4-78c2-4361-8243-e98c35246703-maas-api-openshift-ca\") pod \"23fb2af4-78c2-4361-8243-e98c35246703\" (UID: \"23fb2af4-78c2-4361-8243-e98c35246703\") " Nov 03 15:32:30.537028 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:30.536900 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23fb2af4-78c2-4361-8243-e98c35246703-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23fb2af4-78c2-4361-8243-e98c35246703-maas-api-xks-ca\") pod \"23fb2af4-78c2-4361-8243-e98c35246703\" (UID: \"23fb2af4-78c2-4361-8243-e98c35246703\") " Nov 03 15:32:30.537211 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:30.537191 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23fb2af4-78c2-4361-8243-e98c35246703-kube-api-access-kgdf9\" (UniqueName: \"kubernetes.io/projected/23fb2af4-78c2-4361-8243-e98c35246703-kube-api-access-kgdf9\") pod \"23fb2af4-78c2-4361-8243-e98c35246703\" (UID: \"23fb2af4-78c2-4361-8243-e98c35246703\") " Nov 03 15:32:30.537269 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:30.537230 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23fb2af4-78c2-4361-8243-e98c35246703-maas-api-openshift-ca" pod "23fb2af4-78c2-4361-8243-e98c35246703" (UID: "23fb2af4-78c2-4361-8243-e98c35246703"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:32:30.537758 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:30.537738 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23fb2af4-78c2-4361-8243-e98c35246703-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:30.538931 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:30.538907 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23fb2af4-78c2-4361-8243-e98c35246703-maas-api-xks-ca" pod "23fb2af4-78c2-4361-8243-e98c35246703" (UID: "23fb2af4-78c2-4361-8243-e98c35246703"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:32:30.539143 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:30.539117 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fb2af4-78c2-4361-8243-e98c35246703-kube-api-access-kgdf9" pod "23fb2af4-78c2-4361-8243-e98c35246703" (UID: "23fb2af4-78c2-4361-8243-e98c35246703"). InnerVolumeSpecName "kube-api-access-kgdf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:32:30.639593 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:30.639548 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgdf9\" (UniqueName: \"kubernetes.io/projected/23fb2af4-78c2-4361-8243-e98c35246703-kube-api-access-kgdf9\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:30.639744 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:30.639617 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23fb2af4-78c2-4361-8243-e98c35246703-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:31.299992 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:31.299962 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh" Nov 03 15:32:31.300400 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:31.300004 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh" event={"ID":"23fb2af4-78c2-4361-8243-e98c35246703","Type":"ContainerDied","Data":"27e267ffb04409fb36f224c55f32f8531ecccb32a3683fa9efb6ea611c9576a4"} Nov 03 15:32:31.300400 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:31.300056 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27e267ffb04409fb36f224c55f32f8531ecccb32a3683fa9efb6ea611c9576a4" Nov 03 15:32:34.818992 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:34.818958 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:35.016618 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.016555 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:35.413254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.413220 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v"] Nov 03 15:32:35.413860 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.413846 4557 cpu_manager.go:414] "containerMap: removing container" podUID="23fb2af4-78c2-4361-8243-e98c35246703" containerName="revoke-keys" Nov 03 15:32:35.413903 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.413860 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23fb2af4-78c2-4361-8243-e98c35246703" containerName="revoke-keys" Nov 03 15:32:35.413966 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.413957 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="23fb2af4-78c2-4361-8243-e98c35246703" containerName="revoke-keys" Nov 03 15:32:35.419034 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.419015 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v" Nov 03 15:32:35.473389 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.473360 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:32:35.478472 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.478439 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v"] Nov 03 15:32:35.519290 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.519248 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2e445f-70dc-4678-b24b-e7592a643418-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a6ee148-nv86v\" (UID: \"8d2e445f-70dc-4678-b24b-e7592a643418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v" Nov 03 15:32:35.519501 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.519411 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2e445f-70dc-4678-b24b-e7592a643418-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a6ee148-nv86v\" (UID: \"8d2e445f-70dc-4678-b24b-e7592a643418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v" Nov 03 15:32:35.519501 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.519448 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5cgq\" (UniqueName: \"kubernetes.io/projected/8d2e445f-70dc-4678-b24b-e7592a643418-kube-api-access-h5cgq\") pod \"maas-api-revoke-sub-3a6ee148-nv86v\" (UID: \"8d2e445f-70dc-4678-b24b-e7592a643418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v" Nov 03 15:32:35.521389 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.521362 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:32:35.620836 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.620797 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2e445f-70dc-4678-b24b-e7592a643418-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a6ee148-nv86v\" (UID: \"8d2e445f-70dc-4678-b24b-e7592a643418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v" Nov 03 15:32:35.621005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.620846 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5cgq\" (UniqueName: \"kubernetes.io/projected/8d2e445f-70dc-4678-b24b-e7592a643418-kube-api-access-h5cgq\") pod \"maas-api-revoke-sub-3a6ee148-nv86v\" (UID: \"8d2e445f-70dc-4678-b24b-e7592a643418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v" Nov 03 15:32:35.621090 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.621023 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2e445f-70dc-4678-b24b-e7592a643418-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a6ee148-nv86v\" (UID: \"8d2e445f-70dc-4678-b24b-e7592a643418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v" Nov 03 15:32:35.621698 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.621655 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2e445f-70dc-4678-b24b-e7592a643418-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a6ee148-nv86v\" (UID: \"8d2e445f-70dc-4678-b24b-e7592a643418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v" Nov 03 15:32:35.623157 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.623139 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2e445f-70dc-4678-b24b-e7592a643418-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a6ee148-nv86v\" (UID: \"8d2e445f-70dc-4678-b24b-e7592a643418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v" Nov 03 15:32:35.682649 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.682558 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5cgq\" (UniqueName: \"kubernetes.io/projected/8d2e445f-70dc-4678-b24b-e7592a643418-kube-api-access-h5cgq\") pod \"maas-api-revoke-sub-3a6ee148-nv86v\" (UID: \"8d2e445f-70dc-4678-b24b-e7592a643418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v" Nov 03 15:32:35.732455 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:35.732427 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v" Nov 03 15:32:36.114681 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:36.114641 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v"] Nov 03 15:32:36.115733 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:32:36.115702 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2e445f_70dc_4678_b24b_e7592a643418.slice/crio-b277805336100bc6fd06ab73fcf16ec3021b373caa346a663885d165c278e17d WatchSource:0}: Error finding container b277805336100bc6fd06ab73fcf16ec3021b373caa346a663885d165c278e17d: Status 404 returned error can't find the container with id b277805336100bc6fd06ab73fcf16ec3021b373caa346a663885d165c278e17d Nov 03 15:32:36.338435 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:36.338397 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v" event={"ID":"8d2e445f-70dc-4678-b24b-e7592a643418","Type":"ContainerStarted","Data":"b277805336100bc6fd06ab73fcf16ec3021b373caa346a663885d165c278e17d"} Nov 03 15:32:37.347519 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:37.347435 4557 generic.go:356] "Generic (PLEG): container finished" podID="8d2e445f-70dc-4678-b24b-e7592a643418" containerID="74c5e60261a37f086340b97aca19b1267528ce0afbc1d981eeb1930416ac3f1c" exitCode=0 Nov 03 15:32:37.347913 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:37.347519 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v" event={"ID":"8d2e445f-70dc-4678-b24b-e7592a643418","Type":"ContainerDied","Data":"74c5e60261a37f086340b97aca19b1267528ce0afbc1d981eeb1930416ac3f1c"} Nov 03 15:32:38.524287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:38.524262 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v" Nov 03 15:32:38.675341 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:38.675253 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d2e445f-70dc-4678-b24b-e7592a643418-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2e445f-70dc-4678-b24b-e7592a643418-maas-api-xks-ca\") pod \"8d2e445f-70dc-4678-b24b-e7592a643418\" (UID: \"8d2e445f-70dc-4678-b24b-e7592a643418\") " Nov 03 15:32:38.675341 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:38.675306 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2e445f-70dc-4678-b24b-e7592a643418-kube-api-access-h5cgq\" (UniqueName: \"kubernetes.io/projected/8d2e445f-70dc-4678-b24b-e7592a643418-kube-api-access-h5cgq\") pod \"8d2e445f-70dc-4678-b24b-e7592a643418\" (UID: \"8d2e445f-70dc-4678-b24b-e7592a643418\") " Nov 03 15:32:38.675556 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:38.675373 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d2e445f-70dc-4678-b24b-e7592a643418-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2e445f-70dc-4678-b24b-e7592a643418-maas-api-openshift-ca\") pod \"8d2e445f-70dc-4678-b24b-e7592a643418\" (UID: \"8d2e445f-70dc-4678-b24b-e7592a643418\") " Nov 03 15:32:38.675833 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:38.675796 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2e445f-70dc-4678-b24b-e7592a643418-maas-api-openshift-ca" pod "8d2e445f-70dc-4678-b24b-e7592a643418" (UID: "8d2e445f-70dc-4678-b24b-e7592a643418"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:32:38.676307 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:38.676282 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2e445f-70dc-4678-b24b-e7592a643418-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:38.677641 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:38.677615 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2e445f-70dc-4678-b24b-e7592a643418-kube-api-access-h5cgq" pod "8d2e445f-70dc-4678-b24b-e7592a643418" (UID: "8d2e445f-70dc-4678-b24b-e7592a643418"). InnerVolumeSpecName "kube-api-access-h5cgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:32:38.677904 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:38.677883 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2e445f-70dc-4678-b24b-e7592a643418-maas-api-xks-ca" pod "8d2e445f-70dc-4678-b24b-e7592a643418" (UID: "8d2e445f-70dc-4678-b24b-e7592a643418"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:32:38.777400 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:38.777365 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2e445f-70dc-4678-b24b-e7592a643418-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:38.777400 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:38.777398 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5cgq\" (UniqueName: \"kubernetes.io/projected/8d2e445f-70dc-4678-b24b-e7592a643418-kube-api-access-h5cgq\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:39.366990 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:39.366947 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v" Nov 03 15:32:39.368005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:39.367013 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v" event={"ID":"8d2e445f-70dc-4678-b24b-e7592a643418","Type":"ContainerDied","Data":"b277805336100bc6fd06ab73fcf16ec3021b373caa346a663885d165c278e17d"} Nov 03 15:32:39.368005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:39.367874 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b277805336100bc6fd06ab73fcf16ec3021b373caa346a663885d165c278e17d" Nov 03 15:32:39.445397 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:39.445357 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:40.011379 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:40.011347 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:42.017888 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:42.017855 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:42.220998 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:42.220963 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:45.370819 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:45.370781 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:46.223950 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.223907 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z"] Nov 03 15:32:46.224518 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.224504 4557 cpu_manager.go:414] "containerMap: removing container" podUID="8d2e445f-70dc-4678-b24b-e7592a643418" containerName="revoke-keys" Nov 03 15:32:46.224567 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.224519 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2e445f-70dc-4678-b24b-e7592a643418" containerName="revoke-keys" Nov 03 15:32:46.224660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.224650 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2e445f-70dc-4678-b24b-e7592a643418" containerName="revoke-keys" Nov 03 15:32:46.228525 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.228509 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z" Nov 03 15:32:46.288559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.288529 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z"] Nov 03 15:32:46.289313 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.289287 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31c8374-1162-4974-8d37-b461fdc819a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z\" (UID: \"e31c8374-1162-4974-8d37-b461fdc819a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z" Nov 03 15:32:46.289489 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.289462 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31c8374-1162-4974-8d37-b461fdc819a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z\" (UID: \"e31c8374-1162-4974-8d37-b461fdc819a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z" Nov 03 15:32:46.289734 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.289714 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t7tw\" (UniqueName: \"kubernetes.io/projected/e31c8374-1162-4974-8d37-b461fdc819a6-kube-api-access-6t7tw\") pod \"maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z\" (UID: \"e31c8374-1162-4974-8d37-b461fdc819a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z" Nov 03 15:32:46.331367 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.331335 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:32:46.331367 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.331360 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:32:46.391321 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.391293 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t7tw\" (UniqueName: \"kubernetes.io/projected/e31c8374-1162-4974-8d37-b461fdc819a6-kube-api-access-6t7tw\") pod \"maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z\" (UID: \"e31c8374-1162-4974-8d37-b461fdc819a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z" Nov 03 15:32:46.391783 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.391359 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31c8374-1162-4974-8d37-b461fdc819a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z\" (UID: \"e31c8374-1162-4974-8d37-b461fdc819a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z" Nov 03 15:32:46.391783 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.391427 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31c8374-1162-4974-8d37-b461fdc819a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z\" (UID: \"e31c8374-1162-4974-8d37-b461fdc819a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z" Nov 03 15:32:46.392155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.392129 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31c8374-1162-4974-8d37-b461fdc819a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z\" (UID: \"e31c8374-1162-4974-8d37-b461fdc819a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z" Nov 03 15:32:46.394087 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.394064 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31c8374-1162-4974-8d37-b461fdc819a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z\" (UID: \"e31c8374-1162-4974-8d37-b461fdc819a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z" Nov 03 15:32:46.399247 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.399225 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:46.452999 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.452972 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t7tw\" (UniqueName: \"kubernetes.io/projected/e31c8374-1162-4974-8d37-b461fdc819a6-kube-api-access-6t7tw\") pod \"maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z\" (UID: \"e31c8374-1162-4974-8d37-b461fdc819a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z" Nov 03 15:32:46.530736 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:32:46.530658 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:32:46.543007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.542984 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z" Nov 03 15:32:46.925176 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:32:46.925142 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31c8374_1162_4974_8d37_b461fdc819a6.slice/crio-ebf447cf96af7f9d62d5cccc940b97bea0c4e8e3b0ec8246513e8d3fa88e4833 WatchSource:0}: Error finding container ebf447cf96af7f9d62d5cccc940b97bea0c4e8e3b0ec8246513e8d3fa88e4833: Status 404 returned error can't find the container with id ebf447cf96af7f9d62d5cccc940b97bea0c4e8e3b0ec8246513e8d3fa88e4833 Nov 03 15:32:46.925890 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:46.925865 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z"] Nov 03 15:32:47.433888 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:47.433847 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z" event={"ID":"e31c8374-1162-4974-8d37-b461fdc819a6","Type":"ContainerStarted","Data":"518cbc3bbf7cfa353ed9052a4cecc8b372371420861615edcee99ce3c1cd702d"} Nov 03 15:32:47.434242 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:47.433902 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z" event={"ID":"e31c8374-1162-4974-8d37-b461fdc819a6","Type":"ContainerStarted","Data":"ebf447cf96af7f9d62d5cccc940b97bea0c4e8e3b0ec8246513e8d3fa88e4833"} Nov 03 15:32:47.553944 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:47.553892 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z" podStartSLOduration=1.553878481 podStartE2EDuration="1.553878481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:32:47.551775659 +0000 UTC m=+1477.394897201" watchObservedRunningTime="2026-11-03 15:32:47.553878481 +0000 UTC m=+1477.397000086" Nov 03 15:32:48.443750 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:48.443705 4557 generic.go:356] "Generic (PLEG): container finished" podID="e31c8374-1162-4974-8d37-b461fdc819a6" containerID="518cbc3bbf7cfa353ed9052a4cecc8b372371420861615edcee99ce3c1cd702d" exitCode=0 Nov 03 15:32:48.444179 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:48.443793 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z" event={"ID":"e31c8374-1162-4974-8d37-b461fdc819a6","Type":"ContainerDied","Data":"518cbc3bbf7cfa353ed9052a4cecc8b372371420861615edcee99ce3c1cd702d"} Nov 03 15:32:49.606898 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:49.606871 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z" Nov 03 15:32:49.742109 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:49.742028 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e31c8374-1162-4974-8d37-b461fdc819a6-kube-api-access-6t7tw\" (UniqueName: \"kubernetes.io/projected/e31c8374-1162-4974-8d37-b461fdc819a6-kube-api-access-6t7tw\") pod \"e31c8374-1162-4974-8d37-b461fdc819a6\" (UID: \"e31c8374-1162-4974-8d37-b461fdc819a6\") " Nov 03 15:32:49.742109 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:49.742079 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e31c8374-1162-4974-8d37-b461fdc819a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31c8374-1162-4974-8d37-b461fdc819a6-maas-api-openshift-ca\") pod \"e31c8374-1162-4974-8d37-b461fdc819a6\" (UID: \"e31c8374-1162-4974-8d37-b461fdc819a6\") " Nov 03 15:32:49.742345 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:49.742163 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e31c8374-1162-4974-8d37-b461fdc819a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31c8374-1162-4974-8d37-b461fdc819a6-maas-api-xks-ca\") pod \"e31c8374-1162-4974-8d37-b461fdc819a6\" (UID: \"e31c8374-1162-4974-8d37-b461fdc819a6\") " Nov 03 15:32:49.742554 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:49.742518 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e31c8374-1162-4974-8d37-b461fdc819a6-maas-api-openshift-ca" pod "e31c8374-1162-4974-8d37-b461fdc819a6" (UID: "e31c8374-1162-4974-8d37-b461fdc819a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:32:49.742977 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:49.742952 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31c8374-1162-4974-8d37-b461fdc819a6-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:49.744308 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:49.744284 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e31c8374-1162-4974-8d37-b461fdc819a6-maas-api-xks-ca" pod "e31c8374-1162-4974-8d37-b461fdc819a6" (UID: "e31c8374-1162-4974-8d37-b461fdc819a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:32:49.744675 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:49.744653 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31c8374-1162-4974-8d37-b461fdc819a6-kube-api-access-6t7tw" pod "e31c8374-1162-4974-8d37-b461fdc819a6" (UID: "e31c8374-1162-4974-8d37-b461fdc819a6"). InnerVolumeSpecName "kube-api-access-6t7tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:32:49.844363 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:49.844329 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t7tw\" (UniqueName: \"kubernetes.io/projected/e31c8374-1162-4974-8d37-b461fdc819a6-kube-api-access-6t7tw\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:49.844363 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:49.844356 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31c8374-1162-4974-8d37-b461fdc819a6-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:32:50.460596 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:50.460547 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z" event={"ID":"e31c8374-1162-4974-8d37-b461fdc819a6","Type":"ContainerDied","Data":"ebf447cf96af7f9d62d5cccc940b97bea0c4e8e3b0ec8246513e8d3fa88e4833"} Nov 03 15:32:50.460596 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:50.460589 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z" Nov 03 15:32:50.460868 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:50.460626 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebf447cf96af7f9d62d5cccc940b97bea0c4e8e3b0ec8246513e8d3fa88e4833" Nov 03 15:32:50.660091 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:50.660018 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:52.550873 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:52.550842 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:55.482538 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:55.482420 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:58.053767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:58.053727 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc"] Nov 03 15:32:58.102606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:58.102564 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f550b8c-85mjc"] Nov 03 15:32:58.115564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:58.115530 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:32:58.287123 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:58.287088 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ef56ec-1f91-410b-8fda-53c02e439893" path="/var/lib/kubelet/pods/09ef56ec-1f91-410b-8fda-53c02e439893/volumes" Nov 03 15:32:58.312520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:32:58.312440 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:00.406284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:00.406251 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:04.020679 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:04.020643 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:04.422993 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:04.422952 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:05.329421 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:05.329384 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:10.310673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:10.310640 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:10.465216 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:10.465175 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:33:10.472344 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:10.472318 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:33:12.052465 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:12.052432 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq"] Nov 03 15:33:12.102203 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:12.102173 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63533408-rtqzq"] Nov 03 15:33:12.286618 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:12.286564 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3df0dec4-7e3a-4e0d-8233-09336bcdc899" path="/var/lib/kubelet/pods/3df0dec4-7e3a-4e0d-8233-09336bcdc899/volumes" Nov 03 15:33:14.654604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:14.653395 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:16.226831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:16.226800 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:16.530120 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:33:16.530042 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:33:16.618636 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:16.618546 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:16.829820 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:16.829791 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:16.981238 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:16.981207 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn"] Nov 03 15:33:17.031132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:17.031099 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47b989b3-5jttn"] Nov 03 15:33:17.205052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:17.204961 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:17.437846 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:17.437809 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:17.915051 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:17.915020 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:18.135251 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:18.135220 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:18.288747 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:18.288656 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2fb33ba-ad6a-4bc3-8209-792eff19c786" path="/var/lib/kubelet/pods/e2fb33ba-ad6a-4bc3-8209-792eff19c786/volumes" Nov 03 15:33:18.328413 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:18.328384 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:19.399619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:19.399551 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:19.755838 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:19.755720 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:21.123873 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:21.123835 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:21.684995 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:21.684952 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs"] Nov 03 15:33:21.686007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:21.685982 4557 cpu_manager.go:414] "containerMap: removing container" podUID="e31c8374-1162-4974-8d37-b461fdc819a6" containerName="revoke-keys" Nov 03 15:33:21.686007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:21.686003 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e31c8374-1162-4974-8d37-b461fdc819a6" containerName="revoke-keys" Nov 03 15:33:21.686220 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:21.686133 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="e31c8374-1162-4974-8d37-b461fdc819a6" containerName="revoke-keys" Nov 03 15:33:21.693029 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:21.693002 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs" Nov 03 15:33:21.694207 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:21.694180 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs"] Nov 03 15:33:21.746872 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:21.746781 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:33:21.746872 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:21.746836 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:33:21.796749 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:21.796711 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0797e9bb-32a4-44b8-b205-52470cfecca9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs\" (UID: \"0797e9bb-32a4-44b8-b205-52470cfecca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs" Nov 03 15:33:21.796938 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:21.796814 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0797e9bb-32a4-44b8-b205-52470cfecca9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs\" (UID: \"0797e9bb-32a4-44b8-b205-52470cfecca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs" Nov 03 15:33:21.797173 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:21.797152 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2cst\" (UniqueName: \"kubernetes.io/projected/0797e9bb-32a4-44b8-b205-52470cfecca9-kube-api-access-f2cst\") pod \"maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs\" (UID: \"0797e9bb-32a4-44b8-b205-52470cfecca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs" Nov 03 15:33:21.899015 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:21.898974 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0797e9bb-32a4-44b8-b205-52470cfecca9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs\" (UID: \"0797e9bb-32a4-44b8-b205-52470cfecca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs" Nov 03 15:33:21.899199 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:21.899058 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0797e9bb-32a4-44b8-b205-52470cfecca9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs\" (UID: \"0797e9bb-32a4-44b8-b205-52470cfecca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs" Nov 03 15:33:21.899251 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:21.899228 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2cst\" (UniqueName: \"kubernetes.io/projected/0797e9bb-32a4-44b8-b205-52470cfecca9-kube-api-access-f2cst\") pod \"maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs\" (UID: \"0797e9bb-32a4-44b8-b205-52470cfecca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs" Nov 03 15:33:21.899794 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:21.899767 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0797e9bb-32a4-44b8-b205-52470cfecca9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs\" (UID: \"0797e9bb-32a4-44b8-b205-52470cfecca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs" Nov 03 15:33:21.901366 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:21.901341 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0797e9bb-32a4-44b8-b205-52470cfecca9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs\" (UID: \"0797e9bb-32a4-44b8-b205-52470cfecca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs" Nov 03 15:33:21.962853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:21.962788 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2cst\" (UniqueName: \"kubernetes.io/projected/0797e9bb-32a4-44b8-b205-52470cfecca9-kube-api-access-f2cst\") pod \"maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs\" (UID: \"0797e9bb-32a4-44b8-b205-52470cfecca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs" Nov 03 15:33:22.017049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:22.015750 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs" Nov 03 15:33:22.536274 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:33:22.536247 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0797e9bb_32a4_44b8_b205_52470cfecca9.slice/crio-01d3cc0564465e89962dddde0669f05c0cae436b2dfd28fc6df8943dac928018 WatchSource:0}: Error finding container 01d3cc0564465e89962dddde0669f05c0cae436b2dfd28fc6df8943dac928018: Status 404 returned error can't find the container with id 01d3cc0564465e89962dddde0669f05c0cae436b2dfd28fc6df8943dac928018 Nov 03 15:33:22.537044 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:22.537018 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs"] Nov 03 15:33:22.793818 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:22.793788 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs" event={"ID":"0797e9bb-32a4-44b8-b205-52470cfecca9","Type":"ContainerStarted","Data":"01d3cc0564465e89962dddde0669f05c0cae436b2dfd28fc6df8943dac928018"} Nov 03 15:33:22.871857 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:22.871823 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:23.203449 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:23.203370 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:23.356612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:23.356561 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh"] Nov 03 15:33:23.363702 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:23.363667 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d785c53b-2hgxh"] Nov 03 15:33:23.438751 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:23.437817 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:23.438910 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:23.438843 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8"] Nov 03 15:33:23.439108 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:23.439082 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" podUID="4b57651a-2c8b-4df4-84c2-4f5fba809417" containerName="istio-proxy" containerID="cri-o://f1678df45b3e48ef7df977e943f5a7de56f3a1a07b4d3f144a7661066915863d" gracePeriod=30 Nov 03 15:33:23.631413 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:23.631377 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:23.807753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:23.807718 4557 generic.go:356] "Generic (PLEG): container finished" podID="0797e9bb-32a4-44b8-b205-52470cfecca9" containerID="28e783856b6154307b32d2e8206bb798143c819f24e0a2fc2b12f2ccbbb15c54" exitCode=0 Nov 03 15:33:23.807978 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:23.807812 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs" event={"ID":"0797e9bb-32a4-44b8-b205-52470cfecca9","Type":"ContainerDied","Data":"28e783856b6154307b32d2e8206bb798143c819f24e0a2fc2b12f2ccbbb15c54"} Nov 03 15:33:24.113798 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:24.113768 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:24.286564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:24.286536 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23fb2af4-78c2-4361-8243-e98c35246703" path="/var/lib/kubelet/pods/23fb2af4-78c2-4361-8243-e98c35246703/volumes" Nov 03 15:33:24.298447 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:24.298419 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:24.530623 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:24.530499 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:24.722140 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:24.722104 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:25.012237 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:25.012205 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs" Nov 03 15:33:25.113724 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:25.113645 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0797e9bb-32a4-44b8-b205-52470cfecca9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0797e9bb-32a4-44b8-b205-52470cfecca9-maas-api-openshift-ca\") pod \"0797e9bb-32a4-44b8-b205-52470cfecca9\" (UID: \"0797e9bb-32a4-44b8-b205-52470cfecca9\") " Nov 03 15:33:25.113874 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:25.113792 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0797e9bb-32a4-44b8-b205-52470cfecca9-kube-api-access-f2cst\" (UniqueName: \"kubernetes.io/projected/0797e9bb-32a4-44b8-b205-52470cfecca9-kube-api-access-f2cst\") pod \"0797e9bb-32a4-44b8-b205-52470cfecca9\" (UID: \"0797e9bb-32a4-44b8-b205-52470cfecca9\") " Nov 03 15:33:25.113946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:25.113929 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0797e9bb-32a4-44b8-b205-52470cfecca9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0797e9bb-32a4-44b8-b205-52470cfecca9-maas-api-xks-ca\") pod \"0797e9bb-32a4-44b8-b205-52470cfecca9\" (UID: \"0797e9bb-32a4-44b8-b205-52470cfecca9\") " Nov 03 15:33:25.113988 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:25.113968 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0797e9bb-32a4-44b8-b205-52470cfecca9-maas-api-openshift-ca" pod "0797e9bb-32a4-44b8-b205-52470cfecca9" (UID: "0797e9bb-32a4-44b8-b205-52470cfecca9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:33:25.114535 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:25.114512 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0797e9bb-32a4-44b8-b205-52470cfecca9-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:25.115966 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:25.115934 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0797e9bb-32a4-44b8-b205-52470cfecca9-kube-api-access-f2cst" pod "0797e9bb-32a4-44b8-b205-52470cfecca9" (UID: "0797e9bb-32a4-44b8-b205-52470cfecca9"). InnerVolumeSpecName "kube-api-access-f2cst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:33:25.116060 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:25.115964 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0797e9bb-32a4-44b8-b205-52470cfecca9-maas-api-xks-ca" pod "0797e9bb-32a4-44b8-b205-52470cfecca9" (UID: "0797e9bb-32a4-44b8-b205-52470cfecca9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:33:25.216058 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:25.216024 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0797e9bb-32a4-44b8-b205-52470cfecca9-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:25.216058 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:25.216054 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2cst\" (UniqueName: \"kubernetes.io/projected/0797e9bb-32a4-44b8-b205-52470cfecca9-kube-api-access-f2cst\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:25.827070 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:25.827043 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs" Nov 03 15:33:25.827454 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:25.827044 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs" event={"ID":"0797e9bb-32a4-44b8-b205-52470cfecca9","Type":"ContainerDied","Data":"01d3cc0564465e89962dddde0669f05c0cae436b2dfd28fc6df8943dac928018"} Nov 03 15:33:25.827454 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:25.827176 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01d3cc0564465e89962dddde0669f05c0cae436b2dfd28fc6df8943dac928018" Nov 03 15:33:26.581443 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:26.581412 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:26.789482 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:26.789452 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:27.820379 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:27.820345 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2"] Nov 03 15:33:27.821023 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:27.820528 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" podUID="6735b185-18df-47d2-b6a3-c1ed601d009d" containerName="maas-api" containerID="cri-o://3c684f6bea412de8719b1335705d345f72cfcf103691f1af447d80db94942870" gracePeriod=30 Nov 03 15:33:27.959865 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:27.959832 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:28.173736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.172804 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:33:28.275328 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.275293 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6735b185-18df-47d2-b6a3-c1ed601d009d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6735b185-18df-47d2-b6a3-c1ed601d009d-metrics-certs\") pod \"6735b185-18df-47d2-b6a3-c1ed601d009d\" (UID: \"6735b185-18df-47d2-b6a3-c1ed601d009d\") " Nov 03 15:33:28.275328 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.275331 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6735b185-18df-47d2-b6a3-c1ed601d009d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6735b185-18df-47d2-b6a3-c1ed601d009d-maas-api-tls\") pod \"6735b185-18df-47d2-b6a3-c1ed601d009d\" (UID: \"6735b185-18df-47d2-b6a3-c1ed601d009d\") " Nov 03 15:33:28.275635 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.275559 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6735b185-18df-47d2-b6a3-c1ed601d009d-kube-api-access-c4qdx\" (UniqueName: \"kubernetes.io/projected/6735b185-18df-47d2-b6a3-c1ed601d009d-kube-api-access-c4qdx\") pod \"6735b185-18df-47d2-b6a3-c1ed601d009d\" (UID: \"6735b185-18df-47d2-b6a3-c1ed601d009d\") " Nov 03 15:33:28.275716 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.275659 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6735b185-18df-47d2-b6a3-c1ed601d009d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6735b185-18df-47d2-b6a3-c1ed601d009d-openshift-service-ca\") pod \"6735b185-18df-47d2-b6a3-c1ed601d009d\" (UID: \"6735b185-18df-47d2-b6a3-c1ed601d009d\") " Nov 03 15:33:28.276036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.276009 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6735b185-18df-47d2-b6a3-c1ed601d009d-openshift-service-ca" pod "6735b185-18df-47d2-b6a3-c1ed601d009d" (UID: "6735b185-18df-47d2-b6a3-c1ed601d009d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:33:28.276292 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.276269 4557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6735b185-18df-47d2-b6a3-c1ed601d009d-openshift-service-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:28.278143 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.278105 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6735b185-18df-47d2-b6a3-c1ed601d009d-kube-api-access-c4qdx" pod "6735b185-18df-47d2-b6a3-c1ed601d009d" (UID: "6735b185-18df-47d2-b6a3-c1ed601d009d"). InnerVolumeSpecName "kube-api-access-c4qdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:33:28.278243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.278145 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6735b185-18df-47d2-b6a3-c1ed601d009d-metrics-certs" pod "6735b185-18df-47d2-b6a3-c1ed601d009d" (UID: "6735b185-18df-47d2-b6a3-c1ed601d009d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:33:28.278243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.278156 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6735b185-18df-47d2-b6a3-c1ed601d009d-maas-api-tls" pod "6735b185-18df-47d2-b6a3-c1ed601d009d" (UID: "6735b185-18df-47d2-b6a3-c1ed601d009d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:33:28.377798 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.377721 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4qdx\" (UniqueName: \"kubernetes.io/projected/6735b185-18df-47d2-b6a3-c1ed601d009d-kube-api-access-c4qdx\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:28.377798 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.377752 4557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6735b185-18df-47d2-b6a3-c1ed601d009d-metrics-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:28.377798 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.377761 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6735b185-18df-47d2-b6a3-c1ed601d009d-maas-api-tls\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:28.731032 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.731000 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:33:28.782830 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.782801 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b57651a-2c8b-4df4-84c2-4f5fba809417-kube-api-access-hrd7n\" (UniqueName: \"kubernetes.io/projected/4b57651a-2c8b-4df4-84c2-4f5fba809417-kube-api-access-hrd7n\") pod \"4b57651a-2c8b-4df4-84c2-4f5fba809417\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " Nov 03 15:33:28.783024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.782888 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-credential-socket\") pod \"4b57651a-2c8b-4df4-84c2-4f5fba809417\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " Nov 03 15:33:28.783024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.782932 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-envoy\") pod \"4b57651a-2c8b-4df4-84c2-4f5fba809417\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " Nov 03 15:33:28.783024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.782995 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-workload-socket\") pod \"4b57651a-2c8b-4df4-84c2-4f5fba809417\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " Nov 03 15:33:28.783185 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.783072 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-data\") pod \"4b57651a-2c8b-4df4-84c2-4f5fba809417\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " Nov 03 15:33:28.783185 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.783131 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-podinfo\") pod \"4b57651a-2c8b-4df4-84c2-4f5fba809417\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " Nov 03 15:33:28.783185 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.783162 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-token\" (UniqueName: \"kubernetes.io/projected/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-token\") pod \"4b57651a-2c8b-4df4-84c2-4f5fba809417\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " Nov 03 15:33:28.783185 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.783165 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-credential-socket" pod "4b57651a-2c8b-4df4-84c2-4f5fba809417" (UID: "4b57651a-2c8b-4df4-84c2-4f5fba809417"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:33:28.783382 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.783302 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-workload-certs\") pod \"4b57651a-2c8b-4df4-84c2-4f5fba809417\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " Nov 03 15:33:28.783382 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.783322 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-workload-socket" pod "4b57651a-2c8b-4df4-84c2-4f5fba809417" (UID: "4b57651a-2c8b-4df4-84c2-4f5fba809417"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:33:28.783382 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.783353 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b57651a-2c8b-4df4-84c2-4f5fba809417-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b57651a-2c8b-4df4-84c2-4f5fba809417-istiod-ca-cert\") pod \"4b57651a-2c8b-4df4-84c2-4f5fba809417\" (UID: \"4b57651a-2c8b-4df4-84c2-4f5fba809417\") " Nov 03 15:33:28.783557 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.783375 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-data" pod "4b57651a-2c8b-4df4-84c2-4f5fba809417" (UID: "4b57651a-2c8b-4df4-84c2-4f5fba809417"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:33:28.783639 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.783575 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-workload-certs" pod "4b57651a-2c8b-4df4-84c2-4f5fba809417" (UID: "4b57651a-2c8b-4df4-84c2-4f5fba809417"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:33:28.783867 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.783839 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b57651a-2c8b-4df4-84c2-4f5fba809417-istiod-ca-cert" pod "4b57651a-2c8b-4df4-84c2-4f5fba809417" (UID: "4b57651a-2c8b-4df4-84c2-4f5fba809417"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:33:28.784511 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.784484 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:28.784659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.784516 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:28.784659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.784533 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:28.784659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.784549 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:28.784659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.784562 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b57651a-2c8b-4df4-84c2-4f5fba809417-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:28.785470 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.785439 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b57651a-2c8b-4df4-84c2-4f5fba809417-kube-api-access-hrd7n" pod "4b57651a-2c8b-4df4-84c2-4f5fba809417" (UID: "4b57651a-2c8b-4df4-84c2-4f5fba809417"). InnerVolumeSpecName "kube-api-access-hrd7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:33:28.785604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.785519 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-envoy" pod "4b57651a-2c8b-4df4-84c2-4f5fba809417" (UID: "4b57651a-2c8b-4df4-84c2-4f5fba809417"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:33:28.785604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.785538 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-token" pod "4b57651a-2c8b-4df4-84c2-4f5fba809417" (UID: "4b57651a-2c8b-4df4-84c2-4f5fba809417"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:33:28.785892 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.785867 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-podinfo" pod "4b57651a-2c8b-4df4-84c2-4f5fba809417" (UID: "4b57651a-2c8b-4df4-84c2-4f5fba809417"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:33:28.878659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.877465 4557 generic.go:356] "Generic (PLEG): container finished" podID="4b57651a-2c8b-4df4-84c2-4f5fba809417" containerID="f1678df45b3e48ef7df977e943f5a7de56f3a1a07b4d3f144a7661066915863d" exitCode=0 Nov 03 15:33:28.878659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.877752 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" event={"ID":"4b57651a-2c8b-4df4-84c2-4f5fba809417","Type":"ContainerDied","Data":"f1678df45b3e48ef7df977e943f5a7de56f3a1a07b4d3f144a7661066915863d"} Nov 03 15:33:28.878659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.877861 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" event={"ID":"4b57651a-2c8b-4df4-84c2-4f5fba809417","Type":"ContainerDied","Data":"6400399bb8446061e1e35c6262cf1fac35d078822a9137dafa221a9207b30923"} Nov 03 15:33:28.878659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.877904 4557 scope.go:122] "RemoveContainer" containerID="f1678df45b3e48ef7df977e943f5a7de56f3a1a07b4d3f144a7661066915863d" Nov 03 15:33:28.878659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.878104 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8" Nov 03 15:33:28.893845 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.893813 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrd7n\" (UniqueName: \"kubernetes.io/projected/4b57651a-2c8b-4df4-84c2-4f5fba809417-kube-api-access-hrd7n\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:28.895984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.895965 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:28.896212 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.896169 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:28.896344 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.896331 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b57651a-2c8b-4df4-84c2-4f5fba809417-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:28.896640 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.895644 4557 generic.go:356] "Generic (PLEG): container finished" podID="6735b185-18df-47d2-b6a3-c1ed601d009d" containerID="3c684f6bea412de8719b1335705d345f72cfcf103691f1af447d80db94942870" exitCode=0 Nov 03 15:33:28.896869 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.895675 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" event={"ID":"6735b185-18df-47d2-b6a3-c1ed601d009d","Type":"ContainerDied","Data":"3c684f6bea412de8719b1335705d345f72cfcf103691f1af447d80db94942870"} Nov 03 15:33:28.897033 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.897014 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" event={"ID":"6735b185-18df-47d2-b6a3-c1ed601d009d","Type":"ContainerDied","Data":"faa0a56cf39a246a8bb47b2164c3f6699e32e717e435b01bdcf52bec6762c5c0"} Nov 03 15:33:28.897170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.895847 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2" Nov 03 15:33:28.925322 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.925285 4557 scope.go:122] "RemoveContainer" containerID="e6c9817163d5e9bd3b616de0cc4ce8359718ab7abd15f2e98c58b02312bc73cb" Nov 03 15:33:28.960644 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.960522 4557 scope.go:122] "RemoveContainer" containerID="f1678df45b3e48ef7df977e943f5a7de56f3a1a07b4d3f144a7661066915863d" Nov 03 15:33:28.973606 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:33:28.968107 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1678df45b3e48ef7df977e943f5a7de56f3a1a07b4d3f144a7661066915863d\": container with ID starting with f1678df45b3e48ef7df977e943f5a7de56f3a1a07b4d3f144a7661066915863d not found: ID does not exist" containerID="f1678df45b3e48ef7df977e943f5a7de56f3a1a07b4d3f144a7661066915863d" Nov 03 15:33:28.973606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.968154 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1678df45b3e48ef7df977e943f5a7de56f3a1a07b4d3f144a7661066915863d"} err="failed to get container status \"f1678df45b3e48ef7df977e943f5a7de56f3a1a07b4d3f144a7661066915863d\": rpc error: code = NotFound desc = could not find container \"f1678df45b3e48ef7df977e943f5a7de56f3a1a07b4d3f144a7661066915863d\": container with ID starting with f1678df45b3e48ef7df977e943f5a7de56f3a1a07b4d3f144a7661066915863d not found: ID does not exist" Nov 03 15:33:28.973606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.968182 4557 scope.go:122] "RemoveContainer" containerID="e6c9817163d5e9bd3b616de0cc4ce8359718ab7abd15f2e98c58b02312bc73cb" Nov 03 15:33:28.973606 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:33:28.968522 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6c9817163d5e9bd3b616de0cc4ce8359718ab7abd15f2e98c58b02312bc73cb\": container with ID starting with e6c9817163d5e9bd3b616de0cc4ce8359718ab7abd15f2e98c58b02312bc73cb not found: ID does not exist" containerID="e6c9817163d5e9bd3b616de0cc4ce8359718ab7abd15f2e98c58b02312bc73cb" Nov 03 15:33:28.973606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.968555 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6c9817163d5e9bd3b616de0cc4ce8359718ab7abd15f2e98c58b02312bc73cb"} err="failed to get container status \"e6c9817163d5e9bd3b616de0cc4ce8359718ab7abd15f2e98c58b02312bc73cb\": rpc error: code = NotFound desc = could not find container \"e6c9817163d5e9bd3b616de0cc4ce8359718ab7abd15f2e98c58b02312bc73cb\": container with ID starting with e6c9817163d5e9bd3b616de0cc4ce8359718ab7abd15f2e98c58b02312bc73cb not found: ID does not exist" Nov 03 15:33:28.973606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:28.968575 4557 scope.go:122] "RemoveContainer" containerID="3c684f6bea412de8719b1335705d345f72cfcf103691f1af447d80db94942870" Nov 03 15:33:29.016864 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:29.016831 4557 scope.go:122] "RemoveContainer" containerID="3c684f6bea412de8719b1335705d345f72cfcf103691f1af447d80db94942870" Nov 03 15:33:29.017324 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:33:29.017290 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c684f6bea412de8719b1335705d345f72cfcf103691f1af447d80db94942870\": container with ID starting with 3c684f6bea412de8719b1335705d345f72cfcf103691f1af447d80db94942870 not found: ID does not exist" containerID="3c684f6bea412de8719b1335705d345f72cfcf103691f1af447d80db94942870" Nov 03 15:33:29.017418 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:29.017337 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c684f6bea412de8719b1335705d345f72cfcf103691f1af447d80db94942870"} err="failed to get container status \"3c684f6bea412de8719b1335705d345f72cfcf103691f1af447d80db94942870\": rpc error: code = NotFound desc = could not find container \"3c684f6bea412de8719b1335705d345f72cfcf103691f1af447d80db94942870\": container with ID starting with 3c684f6bea412de8719b1335705d345f72cfcf103691f1af447d80db94942870 not found: ID does not exist" Nov 03 15:33:29.067314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:29.067283 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb"] Nov 03 15:33:29.076289 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:29.076255 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8"] Nov 03 15:33:29.082897 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:29.082867 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7736cce7-fhdbb"] Nov 03 15:33:29.087932 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:29.087897 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-d785c53b-openshift-default-787d6f5445-v95v8"] Nov 03 15:33:29.182325 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:29.182244 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2"] Nov 03 15:33:29.188069 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:29.188032 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cceaf27f-8885f5fbd-57kg2"] Nov 03 15:33:29.688836 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:29.688800 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:29.920983 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:29.920954 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:30.138009 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:30.137969 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:30.226947 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:30.226914 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z"] Nov 03 15:33:30.277103 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:30.277075 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cceaf27f-rp58z"] Nov 03 15:33:30.287212 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:30.287185 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28bef2c6-f2d1-4087-8089-4e15c7d51db2" path="/var/lib/kubelet/pods/28bef2c6-f2d1-4087-8089-4e15c7d51db2/volumes" Nov 03 15:33:30.287766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:30.287751 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b57651a-2c8b-4df4-84c2-4f5fba809417" path="/var/lib/kubelet/pods/4b57651a-2c8b-4df4-84c2-4f5fba809417/volumes" Nov 03 15:33:30.288205 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:30.288192 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6735b185-18df-47d2-b6a3-c1ed601d009d" path="/var/lib/kubelet/pods/6735b185-18df-47d2-b6a3-c1ed601d009d/volumes" Nov 03 15:33:30.288909 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:30.288895 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e31c8374-1162-4974-8d37-b461fdc819a6" path="/var/lib/kubelet/pods/e31c8374-1162-4974-8d37-b461fdc819a6/volumes" Nov 03 15:33:30.989747 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:30.977689 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:31.967925 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:31.967895 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:32.179147 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:32.179117 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:32.943657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:32.943546 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:33.258738 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:33.258661 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk"] Nov 03 15:33:33.376901 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:33.376868 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh"] Nov 03 15:33:33.699953 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:33.699898 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:33.904978 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:33.904940 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:35.203167 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:35.203132 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:38.259518 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.259447 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk" podUID="2c732128-c679-47d6-97aa-c8fcb6daa895" containerName="payload-processing" containerID="cri-o://df440803d779b83dbe8253b85d4340662d2309511dfe7b0b39c95ff04275ac1c" gracePeriod=25 Nov 03 15:33:38.377131 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.377088 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh" podUID="2ecb5a70-d410-4f8c-a6f8-167bc2cc4979" containerName="payload-pre-processing" containerID="cri-o://d97e3a87702b4d876e308d2a609c6bbdc580e854265427106bbc4d21f1cdb423" gracePeriod=25 Nov 03 15:33:38.559909 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.559878 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk" Nov 03 15:33:38.646731 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.646701 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c732128-c679-47d6-97aa-c8fcb6daa895-kube-api-access-7lvjz\" (UniqueName: \"kubernetes.io/projected/2c732128-c679-47d6-97aa-c8fcb6daa895-kube-api-access-7lvjz\") pod \"2c732128-c679-47d6-97aa-c8fcb6daa895\" (UID: \"2c732128-c679-47d6-97aa-c8fcb6daa895\") " Nov 03 15:33:38.646980 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.646960 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c732128-c679-47d6-97aa-c8fcb6daa895-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c732128-c679-47d6-97aa-c8fcb6daa895-plugins-config-volume\") pod \"2c732128-c679-47d6-97aa-c8fcb6daa895\" (UID: \"2c732128-c679-47d6-97aa-c8fcb6daa895\") " Nov 03 15:33:38.647350 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.647324 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c732128-c679-47d6-97aa-c8fcb6daa895-plugins-config-volume" pod "2c732128-c679-47d6-97aa-c8fcb6daa895" (UID: "2c732128-c679-47d6-97aa-c8fcb6daa895"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:33:38.647529 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.647513 4557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c732128-c679-47d6-97aa-c8fcb6daa895-plugins-config-volume\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:38.649012 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.648979 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c732128-c679-47d6-97aa-c8fcb6daa895-kube-api-access-7lvjz" pod "2c732128-c679-47d6-97aa-c8fcb6daa895" (UID: "2c732128-c679-47d6-97aa-c8fcb6daa895"). InnerVolumeSpecName "kube-api-access-7lvjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:33:38.670469 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.670449 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh" Nov 03 15:33:38.748513 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.748472 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ecb5a70-d410-4f8c-a6f8-167bc2cc4979-kube-api-access-l6w7f\" (UniqueName: \"kubernetes.io/projected/2ecb5a70-d410-4f8c-a6f8-167bc2cc4979-kube-api-access-l6w7f\") pod \"2ecb5a70-d410-4f8c-a6f8-167bc2cc4979\" (UID: \"2ecb5a70-d410-4f8c-a6f8-167bc2cc4979\") " Nov 03 15:33:38.748718 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.748705 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ecb5a70-d410-4f8c-a6f8-167bc2cc4979-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb5a70-d410-4f8c-a6f8-167bc2cc4979-plugins-config-volume\") pod \"2ecb5a70-d410-4f8c-a6f8-167bc2cc4979\" (UID: \"2ecb5a70-d410-4f8c-a6f8-167bc2cc4979\") " Nov 03 15:33:38.749033 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.749015 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lvjz\" (UniqueName: \"kubernetes.io/projected/2c732128-c679-47d6-97aa-c8fcb6daa895-kube-api-access-7lvjz\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:38.749033 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.749024 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ecb5a70-d410-4f8c-a6f8-167bc2cc4979-plugins-config-volume" pod "2ecb5a70-d410-4f8c-a6f8-167bc2cc4979" (UID: "2ecb5a70-d410-4f8c-a6f8-167bc2cc4979"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:33:38.750471 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.750444 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecb5a70-d410-4f8c-a6f8-167bc2cc4979-kube-api-access-l6w7f" pod "2ecb5a70-d410-4f8c-a6f8-167bc2cc4979" (UID: "2ecb5a70-d410-4f8c-a6f8-167bc2cc4979"). InnerVolumeSpecName "kube-api-access-l6w7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:33:38.850570 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.850541 4557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb5a70-d410-4f8c-a6f8-167bc2cc4979-plugins-config-volume\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:38.850570 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.850568 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6w7f\" (UniqueName: \"kubernetes.io/projected/2ecb5a70-d410-4f8c-a6f8-167bc2cc4979-kube-api-access-l6w7f\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:38.988457 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.988426 4557 generic.go:356] "Generic (PLEG): container finished" podID="2c732128-c679-47d6-97aa-c8fcb6daa895" containerID="df440803d779b83dbe8253b85d4340662d2309511dfe7b0b39c95ff04275ac1c" exitCode=0 Nov 03 15:33:38.988660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.988467 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk" Nov 03 15:33:38.988660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.988503 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk" event={"ID":"2c732128-c679-47d6-97aa-c8fcb6daa895","Type":"ContainerDied","Data":"df440803d779b83dbe8253b85d4340662d2309511dfe7b0b39c95ff04275ac1c"} Nov 03 15:33:38.988660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.988562 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk" event={"ID":"2c732128-c679-47d6-97aa-c8fcb6daa895","Type":"ContainerDied","Data":"8c955a827120ca8c2343796a2f8976e83d8f2433f3c1309906617082fe050570"} Nov 03 15:33:38.988660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.988604 4557 scope.go:122] "RemoveContainer" containerID="df440803d779b83dbe8253b85d4340662d2309511dfe7b0b39c95ff04275ac1c" Nov 03 15:33:38.990314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.990290 4557 generic.go:356] "Generic (PLEG): container finished" podID="2ecb5a70-d410-4f8c-a6f8-167bc2cc4979" containerID="d97e3a87702b4d876e308d2a609c6bbdc580e854265427106bbc4d21f1cdb423" exitCode=0 Nov 03 15:33:38.990451 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.990346 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh" Nov 03 15:33:38.990451 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.990375 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh" event={"ID":"2ecb5a70-d410-4f8c-a6f8-167bc2cc4979","Type":"ContainerDied","Data":"d97e3a87702b4d876e308d2a609c6bbdc580e854265427106bbc4d21f1cdb423"} Nov 03 15:33:38.990451 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.990428 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh" event={"ID":"2ecb5a70-d410-4f8c-a6f8-167bc2cc4979","Type":"ContainerDied","Data":"a0c5f758454b24ccbea28359ae04c87e33cdb2a92d2f13eae73d8c85cf05c18f"} Nov 03 15:33:38.999525 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.999504 4557 scope.go:122] "RemoveContainer" containerID="df440803d779b83dbe8253b85d4340662d2309511dfe7b0b39c95ff04275ac1c" Nov 03 15:33:38.999850 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:33:38.999816 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df440803d779b83dbe8253b85d4340662d2309511dfe7b0b39c95ff04275ac1c\": container with ID starting with df440803d779b83dbe8253b85d4340662d2309511dfe7b0b39c95ff04275ac1c not found: ID does not exist" containerID="df440803d779b83dbe8253b85d4340662d2309511dfe7b0b39c95ff04275ac1c" Nov 03 15:33:38.999950 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.999858 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df440803d779b83dbe8253b85d4340662d2309511dfe7b0b39c95ff04275ac1c"} err="failed to get container status \"df440803d779b83dbe8253b85d4340662d2309511dfe7b0b39c95ff04275ac1c\": rpc error: code = NotFound desc = could not find container \"df440803d779b83dbe8253b85d4340662d2309511dfe7b0b39c95ff04275ac1c\": container with ID starting with df440803d779b83dbe8253b85d4340662d2309511dfe7b0b39c95ff04275ac1c not found: ID does not exist" Nov 03 15:33:38.999950 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:38.999875 4557 scope.go:122] "RemoveContainer" containerID="d97e3a87702b4d876e308d2a609c6bbdc580e854265427106bbc4d21f1cdb423" Nov 03 15:33:39.010098 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:39.010077 4557 scope.go:122] "RemoveContainer" containerID="d97e3a87702b4d876e308d2a609c6bbdc580e854265427106bbc4d21f1cdb423" Nov 03 15:33:39.010368 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:33:39.010343 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97e3a87702b4d876e308d2a609c6bbdc580e854265427106bbc4d21f1cdb423\": container with ID starting with d97e3a87702b4d876e308d2a609c6bbdc580e854265427106bbc4d21f1cdb423 not found: ID does not exist" containerID="d97e3a87702b4d876e308d2a609c6bbdc580e854265427106bbc4d21f1cdb423" Nov 03 15:33:39.010414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:39.010379 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97e3a87702b4d876e308d2a609c6bbdc580e854265427106bbc4d21f1cdb423"} err="failed to get container status \"d97e3a87702b4d876e308d2a609c6bbdc580e854265427106bbc4d21f1cdb423\": rpc error: code = NotFound desc = could not find container \"d97e3a87702b4d876e308d2a609c6bbdc580e854265427106bbc4d21f1cdb423\": container with ID starting with d97e3a87702b4d876e308d2a609c6bbdc580e854265427106bbc4d21f1cdb423 not found: ID does not exist" Nov 03 15:33:39.051750 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:39.051543 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng"] Nov 03 15:33:39.056593 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:39.056557 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0addb6-hwbng"] Nov 03 15:33:39.156554 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:39.156464 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh"] Nov 03 15:33:39.160735 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:39.160713 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-77d0e91e-6794cdb759-8cmlh"] Nov 03 15:33:39.270318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:39.270289 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk"] Nov 03 15:33:39.279284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:39.279264 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77d0e91e-6bd69f5479-bjgzk"] Nov 03 15:33:40.215142 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:40.215103 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:40.286722 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:40.286692 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c732128-c679-47d6-97aa-c8fcb6daa895" path="/var/lib/kubelet/pods/2c732128-c679-47d6-97aa-c8fcb6daa895/volumes" Nov 03 15:33:40.287159 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:40.287145 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ecb5a70-d410-4f8c-a6f8-167bc2cc4979" path="/var/lib/kubelet/pods/2ecb5a70-d410-4f8c-a6f8-167bc2cc4979/volumes" Nov 03 15:33:40.287550 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:40.287537 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7fceb3c-3f88-4f23-9387-099c4287f353" path="/var/lib/kubelet/pods/f7fceb3c-3f88-4f23-9387-099c4287f353/volumes" Nov 03 15:33:44.050658 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:44.050627 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt"] Nov 03 15:33:44.100971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:44.100943 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa234c5e-brglt"] Nov 03 15:33:44.286768 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:44.286733 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0" path="/var/lib/kubelet/pods/b8ba5d83-546e-4eaa-b14e-7ffd7abfa3b0/volumes" Nov 03 15:33:45.120296 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:45.120261 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:46.535802 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:33:46.535737 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:33:50.220823 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:50.220788 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:52.051685 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:52.051653 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp"] Nov 03 15:33:52.101212 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:52.101176 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db5e2ad8-kqdnp"] Nov 03 15:33:52.287026 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:52.286995 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eca2f9ef-6560-4b12-a003-1980889cec73" path="/var/lib/kubelet/pods/eca2f9ef-6560-4b12-a003-1980889cec73/volumes" Nov 03 15:33:54.360819 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.360783 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8"] Nov 03 15:33:54.361351 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.361337 4557 cpu_manager.go:414] "containerMap: removing container" podUID="4b57651a-2c8b-4df4-84c2-4f5fba809417" containerName="istio-proxy" Nov 03 15:33:54.361397 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.361352 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b57651a-2c8b-4df4-84c2-4f5fba809417" containerName="istio-proxy" Nov 03 15:33:54.361397 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.361367 4557 cpu_manager.go:414] "containerMap: removing container" podUID="4b57651a-2c8b-4df4-84c2-4f5fba809417" containerName="istio-proxy" Nov 03 15:33:54.361397 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.361373 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b57651a-2c8b-4df4-84c2-4f5fba809417" containerName="istio-proxy" Nov 03 15:33:54.361397 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.361387 4557 cpu_manager.go:414] "containerMap: removing container" podUID="2ecb5a70-d410-4f8c-a6f8-167bc2cc4979" containerName="payload-pre-processing" Nov 03 15:33:54.361397 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.361392 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ecb5a70-d410-4f8c-a6f8-167bc2cc4979" containerName="payload-pre-processing" Nov 03 15:33:54.361619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.361405 4557 cpu_manager.go:414] "containerMap: removing container" podUID="6735b185-18df-47d2-b6a3-c1ed601d009d" containerName="maas-api" Nov 03 15:33:54.361619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.361410 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6735b185-18df-47d2-b6a3-c1ed601d009d" containerName="maas-api" Nov 03 15:33:54.361619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.361422 4557 cpu_manager.go:414] "containerMap: removing container" podUID="2c732128-c679-47d6-97aa-c8fcb6daa895" containerName="payload-processing" Nov 03 15:33:54.361619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.361427 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c732128-c679-47d6-97aa-c8fcb6daa895" containerName="payload-processing" Nov 03 15:33:54.361619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.361439 4557 cpu_manager.go:414] "containerMap: removing container" podUID="0797e9bb-32a4-44b8-b205-52470cfecca9" containerName="revoke-keys" Nov 03 15:33:54.361619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.361443 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0797e9bb-32a4-44b8-b205-52470cfecca9" containerName="revoke-keys" Nov 03 15:33:54.361619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.361526 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ecb5a70-d410-4f8c-a6f8-167bc2cc4979" containerName="payload-pre-processing" Nov 03 15:33:54.361619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.361535 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="6735b185-18df-47d2-b6a3-c1ed601d009d" containerName="maas-api" Nov 03 15:33:54.361619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.361543 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c732128-c679-47d6-97aa-c8fcb6daa895" containerName="payload-processing" Nov 03 15:33:54.361619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.361548 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b57651a-2c8b-4df4-84c2-4f5fba809417" containerName="istio-proxy" Nov 03 15:33:54.361619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.361556 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="0797e9bb-32a4-44b8-b205-52470cfecca9" containerName="revoke-keys" Nov 03 15:33:54.361619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.361561 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b57651a-2c8b-4df4-84c2-4f5fba809417" containerName="istio-proxy" Nov 03 15:33:54.366664 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.366644 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.419830 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.419794 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 15:33:54.419830 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.419803 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 15:33:54.420032 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.419803 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 15:33:54.420032 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.419880 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-th69q\"" Nov 03 15:33:54.441504 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.441480 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.441673 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.441656 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.441724 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.441685 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.441760 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.441738 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.441864 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.441834 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.441982 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.441909 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.442039 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.442026 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62glw\" (UniqueName: \"kubernetes.io/projected/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-kube-api-access-62glw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.488620 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.488591 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8"] Nov 03 15:33:54.543292 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.543264 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.543442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.543298 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.543442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.543340 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.543442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.543362 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.543442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.543395 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.543617 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.543506 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62glw\" (UniqueName: \"kubernetes.io/projected/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-kube-api-access-62glw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.543617 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.543559 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.544377 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.544345 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.545937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.545907 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.546063 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.545937 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.546063 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.545969 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.546063 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.545981 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.604546 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.604516 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.604761 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.604741 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62glw\" (UniqueName: \"kubernetes.io/projected/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-kube-api-access-62glw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2scp8\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:54.679190 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:54.679112 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:33:55.110982 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:55.109888 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:33:55.113060 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:55.113037 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8"] Nov 03 15:33:55.113293 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:33:55.113270 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa511cdc_f93b_4ec6_b490_f11eb1aa5b0e.slice/crio-96bbc3596340270883aaa783bbd09cb33f0b72fb596f76d8d901c85b5fa643a2 WatchSource:0}: Error finding container 96bbc3596340270883aaa783bbd09cb33f0b72fb596f76d8d901c85b5fa643a2: Status 404 returned error can't find the container with id 96bbc3596340270883aaa783bbd09cb33f0b72fb596f76d8d901c85b5fa643a2 Nov 03 15:33:55.204412 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:55.204370 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" event={"ID":"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e","Type":"ContainerStarted","Data":"96bbc3596340270883aaa783bbd09cb33f0b72fb596f76d8d901c85b5fa643a2"} Nov 03 15:33:59.134612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:59.134564 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:33:59.134916 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:59.134661 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:33:59.179095 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:59.178883 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x"] Nov 03 15:33:59.179190 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:59.179119 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" podUID="79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32" containerName="maas-api" containerID="cri-o://faadc153354e149f9ce269bc5660da0b5cc7486ccc5cf0e6308cb665bc2736d5" gracePeriod=30 Nov 03 15:33:59.543478 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:59.543432 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:33:59.621852 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:59.621821 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-metrics-certs\") pod \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\" (UID: \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\") " Nov 03 15:33:59.622168 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:59.621879 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-maas-api-tls\") pod \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\" (UID: \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\") " Nov 03 15:33:59.622168 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:59.621933 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-openshift-service-ca\") pod \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\" (UID: \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\") " Nov 03 15:33:59.622168 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:59.622039 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-kube-api-access-j5dvm\" (UniqueName: \"kubernetes.io/projected/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-kube-api-access-j5dvm\") pod \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\" (UID: \"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32\") " Nov 03 15:33:59.622351 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:59.622331 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-openshift-service-ca" pod "79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32" (UID: "79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:33:59.622777 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:59.622756 4557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-openshift-service-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:59.624480 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:59.624434 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-metrics-certs" pod "79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32" (UID: "79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:33:59.624613 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:59.624520 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-maas-api-tls" pod "79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32" (UID: "79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:33:59.624613 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:59.624550 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-kube-api-access-j5dvm" pod "79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32" (UID: "79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32"). InnerVolumeSpecName "kube-api-access-j5dvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:33:59.724905 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:59.724871 4557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-metrics-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:59.724905 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:59.724905 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-maas-api-tls\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:33:59.725171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:33:59.724921 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5dvm\" (UniqueName: \"kubernetes.io/projected/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32-kube-api-access-j5dvm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:34:00.254667 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:00.254630 4557 generic.go:356] "Generic (PLEG): container finished" podID="79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32" containerID="faadc153354e149f9ce269bc5660da0b5cc7486ccc5cf0e6308cb665bc2736d5" exitCode=0 Nov 03 15:34:00.255192 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:00.254669 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" event={"ID":"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32","Type":"ContainerDied","Data":"faadc153354e149f9ce269bc5660da0b5cc7486ccc5cf0e6308cb665bc2736d5"} Nov 03 15:34:00.255192 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:00.254694 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" Nov 03 15:34:00.255192 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:00.254729 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x" event={"ID":"79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32","Type":"ContainerDied","Data":"5a69587e4630f945fcb9f4173470900552c0b4b7544ff8152cd60369316c7300"} Nov 03 15:34:00.255192 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:00.254817 4557 scope.go:122] "RemoveContainer" containerID="faadc153354e149f9ce269bc5660da0b5cc7486ccc5cf0e6308cb665bc2736d5" Nov 03 15:34:00.264383 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:00.263446 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" event={"ID":"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e","Type":"ContainerStarted","Data":"84919aa5aaee1f580058d126cd4e970b344c99184010c22b6ac436b9a84426fc"} Nov 03 15:34:00.264383 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:00.263643 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:34:00.266340 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:00.265724 4557 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2scp8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:34:00.266340 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:00.265769 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" podUID="fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:34:00.279117 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:00.279086 4557 scope.go:122] "RemoveContainer" containerID="faadc153354e149f9ce269bc5660da0b5cc7486ccc5cf0e6308cb665bc2736d5" Nov 03 15:34:00.279461 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:34:00.279440 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faadc153354e149f9ce269bc5660da0b5cc7486ccc5cf0e6308cb665bc2736d5\": container with ID starting with faadc153354e149f9ce269bc5660da0b5cc7486ccc5cf0e6308cb665bc2736d5 not found: ID does not exist" containerID="faadc153354e149f9ce269bc5660da0b5cc7486ccc5cf0e6308cb665bc2736d5" Nov 03 15:34:00.279536 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:00.279473 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faadc153354e149f9ce269bc5660da0b5cc7486ccc5cf0e6308cb665bc2736d5"} err="failed to get container status \"faadc153354e149f9ce269bc5660da0b5cc7486ccc5cf0e6308cb665bc2736d5\": rpc error: code = NotFound desc = could not find container \"faadc153354e149f9ce269bc5660da0b5cc7486ccc5cf0e6308cb665bc2736d5\": container with ID starting with faadc153354e149f9ce269bc5660da0b5cc7486ccc5cf0e6308cb665bc2736d5 not found: ID does not exist" Nov 03 15:34:00.442757 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:00.442702 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" podStartSLOduration=2.425587055 podStartE2EDuration="6.442686254s" totalImagesPullingTime="4.017099199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:34:00.438604261 +0000 UTC m=+1550.281725804" watchObservedRunningTime="2026-11-03 15:34:00.442686254 +0000 UTC m=+1550.285807799" Nov 03 15:34:00.560141 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:00.560107 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x"] Nov 03 15:34:00.566154 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:00.566128 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77d0e91e-656b68ff95-sr67x"] Nov 03 15:34:00.988694 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:00.988551 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:01.186891 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:01.186859 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:01.275682 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:01.275263 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:34:01.384466 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:01.384435 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs"] Nov 03 15:34:01.388802 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:01.388777 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77d0e91e-n5sjs"] Nov 03 15:34:02.050846 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:02.050813 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph"] Nov 03 15:34:02.054845 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:02.054822 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-763c22db-vn5ph"] Nov 03 15:34:02.285923 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:02.285891 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0797e9bb-32a4-44b8-b205-52470cfecca9" path="/var/lib/kubelet/pods/0797e9bb-32a4-44b8-b205-52470cfecca9/volumes" Nov 03 15:34:02.286384 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:02.286369 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e32b0d-6883-42fd-96a4-4a966ded53f4" path="/var/lib/kubelet/pods/18e32b0d-6883-42fd-96a4-4a966ded53f4/volumes" Nov 03 15:34:02.286826 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:02.286808 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32" path="/var/lib/kubelet/pods/79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32/volumes" Nov 03 15:34:03.107839 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:03.107798 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:04.085533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:04.085484 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:04.294140 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:04.294070 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:04.500400 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:04.500304 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:04.697924 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:04.697892 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:05.902714 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:05.902680 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:06.331328 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:06.331295 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:06.520241 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:06.520203 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:07.053361 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:07.053329 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m"] Nov 03 15:34:07.103446 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:07.103414 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f7302e-hk98m"] Nov 03 15:34:08.050549 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:08.050517 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl"] Nov 03 15:34:08.102361 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:08.102330 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bce079c1-zrbbl"] Nov 03 15:34:08.291825 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:08.291779 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e538a54-9596-4967-bfdb-a1acc7fe6aa5" path="/var/lib/kubelet/pods/1e538a54-9596-4967-bfdb-a1acc7fe6aa5/volumes" Nov 03 15:34:08.292533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:08.292510 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5" path="/var/lib/kubelet/pods/9e0f07ad-9dc3-4251-b7b4-8a9d676f11b5/volumes" Nov 03 15:34:10.588319 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:10.588287 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:10.778795 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:10.778761 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:10.996098 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:10.996018 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:11.187550 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:11.187520 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:15.123446 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:15.123415 4557 scope.go:122] "RemoveContainer" containerID="1002bd399aefeb5aa7c2bf3d1b54a9829348cea27b763019e99921eb9e9f2645" Nov 03 15:34:15.133843 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:15.133821 4557 scope.go:122] "RemoveContainer" containerID="b6838e8c3ac2a0c614b49cbfe3373f14d2dd2d3744c6b288486dbfb17f8c5554" Nov 03 15:34:16.037661 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:16.037631 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:16.530784 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:34:16.530744 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:34:16.713855 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:16.713826 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:17.052618 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:17.052571 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd"] Nov 03 15:34:17.056505 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:17.056484 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75040bd4-blpsd"] Nov 03 15:34:17.383788 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:17.383692 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:17.597291 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:17.597003 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:18.133987 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:18.133952 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:18.287698 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:18.287664 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb75610c-d1f1-43aa-a0f0-27c3c438aed2" path="/var/lib/kubelet/pods/fb75610c-d1f1-43aa-a0f0-27c3c438aed2/volumes" Nov 03 15:34:18.339874 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:18.339836 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:20.935519 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:20.935479 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:21.050515 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:21.050478 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k"] Nov 03 15:34:21.099421 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:21.099383 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29eab3f2-kgj7k"] Nov 03 15:34:22.136337 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:22.136301 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:22.286421 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:22.286388 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a" path="/var/lib/kubelet/pods/ddf4b5f3-ecd6-44a3-bf3d-b17bc716071a/volumes" Nov 03 15:34:22.327914 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:22.327876 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:25.389882 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:25.389754 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:25.584263 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:25.584210 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:26.061498 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:26.061455 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9"] Nov 03 15:34:26.106779 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:26.106744 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663878af-lfnw9"] Nov 03 15:34:26.288453 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:26.288400 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ace764-2928-4065-be34-16d73be85112" path="/var/lib/kubelet/pods/73ace764-2928-4065-be34-16d73be85112/volumes" Nov 03 15:34:26.477212 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:26.477129 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:30.591166 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:30.591127 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:31.290032 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:31.289961 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:31.564412 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:31.564365 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:34.050073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:34.050038 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c"] Nov 03 15:34:34.099741 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:34.099713 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26134315-4zj5c"] Nov 03 15:34:34.286110 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:34.286070 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e860c079-fab1-4d54-b509-fe47cb98d018" path="/var/lib/kubelet/pods/e860c079-fab1-4d54-b509-fe47cb98d018/volumes" Nov 03 15:34:35.050728 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:35.050693 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll"] Nov 03 15:34:35.100954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:35.100915 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6192a7dc-9bkll"] Nov 03 15:34:36.286217 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:36.286181 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae" path="/var/lib/kubelet/pods/6370e1ab-ec68-4de8-b8a0-7b9cb3a75aae/volumes" Nov 03 15:34:36.320173 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:36.320141 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:37.624264 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:37.624231 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:39.610197 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:39.610164 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:39.816778 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:39.816740 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:41.876159 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:41.876122 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:46.529811 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:34:46.529768 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:34:47.003707 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:47.003671 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:51.854023 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:51.853926 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:54.062955 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:54.062919 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd"] Nov 03 15:34:54.111721 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:54.111692 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d952a5ad-pqvpd"] Nov 03 15:34:54.286847 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:54.286813 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d8e942a-184c-42d3-aa1b-660ca6884b66" path="/var/lib/kubelet/pods/0d8e942a-184c-42d3-aa1b-660ca6884b66/volumes" Nov 03 15:34:56.397685 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:56.397641 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:34:58.051480 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:58.051450 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2"] Nov 03 15:34:58.101147 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:58.101119 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97de7d34-7zbx2"] Nov 03 15:34:58.294370 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:34:58.294329 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d24b30eb-e757-4180-883d-718ce48bf780" path="/var/lib/kubelet/pods/d24b30eb-e757-4180-883d-718ce48bf780/volumes" Nov 03 15:35:01.116022 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.115987 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf"] Nov 03 15:35:01.116623 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.116600 4557 cpu_manager.go:414] "containerMap: removing container" podUID="79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32" containerName="maas-api" Nov 03 15:35:01.116623 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.116617 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32" containerName="maas-api" Nov 03 15:35:01.116742 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.116718 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ffc7d6-69ab-4c8f-9773-aa0a4daf7e32" containerName="maas-api" Nov 03 15:35:01.120021 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.120006 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf" Nov 03 15:35:01.173497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.173456 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:35:01.177199 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.177175 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf"] Nov 03 15:35:01.222822 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.222784 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:35:01.236293 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.236246 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19da494-26d3-4818-8a32-4af50c7db2d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf\" (UID: \"f19da494-26d3-4818-8a32-4af50c7db2d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf" Nov 03 15:35:01.236542 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.236464 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19da494-26d3-4818-8a32-4af50c7db2d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf\" (UID: \"f19da494-26d3-4818-8a32-4af50c7db2d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf" Nov 03 15:35:01.236684 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.236629 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw74j\" (UniqueName: \"kubernetes.io/projected/f19da494-26d3-4818-8a32-4af50c7db2d2-kube-api-access-lw74j\") pod \"maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf\" (UID: \"f19da494-26d3-4818-8a32-4af50c7db2d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf" Nov 03 15:35:01.338011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.337957 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19da494-26d3-4818-8a32-4af50c7db2d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf\" (UID: \"f19da494-26d3-4818-8a32-4af50c7db2d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf" Nov 03 15:35:01.338227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.338129 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19da494-26d3-4818-8a32-4af50c7db2d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf\" (UID: \"f19da494-26d3-4818-8a32-4af50c7db2d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf" Nov 03 15:35:01.338227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.338223 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw74j\" (UniqueName: \"kubernetes.io/projected/f19da494-26d3-4818-8a32-4af50c7db2d2-kube-api-access-lw74j\") pod \"maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf\" (UID: \"f19da494-26d3-4818-8a32-4af50c7db2d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf" Nov 03 15:35:01.338723 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.338699 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19da494-26d3-4818-8a32-4af50c7db2d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf\" (UID: \"f19da494-26d3-4818-8a32-4af50c7db2d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf" Nov 03 15:35:01.340522 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.340499 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19da494-26d3-4818-8a32-4af50c7db2d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf\" (UID: \"f19da494-26d3-4818-8a32-4af50c7db2d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf" Nov 03 15:35:01.400012 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.399945 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw74j\" (UniqueName: \"kubernetes.io/projected/f19da494-26d3-4818-8a32-4af50c7db2d2-kube-api-access-lw74j\") pod \"maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf\" (UID: \"f19da494-26d3-4818-8a32-4af50c7db2d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf" Nov 03 15:35:01.433816 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.433782 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf" Nov 03 15:35:01.613639 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.613556 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:01.821790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.821757 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf"] Nov 03 15:35:01.822555 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:35:01.822530 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19da494_26d3_4818_8a32_4af50c7db2d2.slice/crio-c23a0826a5a4b45973f18dc5a56afacd746c890d42da680f28a3c9f5762ec45b WatchSource:0}: Error finding container c23a0826a5a4b45973f18dc5a56afacd746c890d42da680f28a3c9f5762ec45b: Status 404 returned error can't find the container with id c23a0826a5a4b45973f18dc5a56afacd746c890d42da680f28a3c9f5762ec45b Nov 03 15:35:01.875754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:01.875721 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf" event={"ID":"f19da494-26d3-4818-8a32-4af50c7db2d2","Type":"ContainerStarted","Data":"c23a0826a5a4b45973f18dc5a56afacd746c890d42da680f28a3c9f5762ec45b"} Nov 03 15:35:02.053246 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:02.053208 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq"] Nov 03 15:35:02.057109 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:02.057083 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df413e8-vxthq"] Nov 03 15:35:02.060634 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:02.060609 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62"] Nov 03 15:35:02.102859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:02.102834 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-387e6deb-nrm62"] Nov 03 15:35:02.286406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:02.286368 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76e93910-04b0-4f0e-99c1-f99b1b444b14" path="/var/lib/kubelet/pods/76e93910-04b0-4f0e-99c1-f99b1b444b14/volumes" Nov 03 15:35:02.286924 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:02.286908 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a0b8bc-24a3-4450-82ca-bbecd615988b" path="/var/lib/kubelet/pods/87a0b8bc-24a3-4450-82ca-bbecd615988b/volumes" Nov 03 15:35:02.885401 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:02.885363 4557 generic.go:356] "Generic (PLEG): container finished" podID="f19da494-26d3-4818-8a32-4af50c7db2d2" containerID="0a44467e8abef767cca147f38eec0432eee4ea8cf5c440a3285aae001906e410" exitCode=0 Nov 03 15:35:02.885596 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:02.885444 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf" event={"ID":"f19da494-26d3-4818-8a32-4af50c7db2d2","Type":"ContainerDied","Data":"0a44467e8abef767cca147f38eec0432eee4ea8cf5c440a3285aae001906e410"} Nov 03 15:35:04.063534 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:04.063506 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf" Nov 03 15:35:04.182532 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:04.182439 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19da494-26d3-4818-8a32-4af50c7db2d2-kube-api-access-lw74j\" (UniqueName: \"kubernetes.io/projected/f19da494-26d3-4818-8a32-4af50c7db2d2-kube-api-access-lw74j\") pod \"f19da494-26d3-4818-8a32-4af50c7db2d2\" (UID: \"f19da494-26d3-4818-8a32-4af50c7db2d2\") " Nov 03 15:35:04.182733 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:04.182572 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19da494-26d3-4818-8a32-4af50c7db2d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19da494-26d3-4818-8a32-4af50c7db2d2-maas-api-openshift-ca\") pod \"f19da494-26d3-4818-8a32-4af50c7db2d2\" (UID: \"f19da494-26d3-4818-8a32-4af50c7db2d2\") " Nov 03 15:35:04.182733 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:04.182699 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f19da494-26d3-4818-8a32-4af50c7db2d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19da494-26d3-4818-8a32-4af50c7db2d2-maas-api-xks-ca\") pod \"f19da494-26d3-4818-8a32-4af50c7db2d2\" (UID: \"f19da494-26d3-4818-8a32-4af50c7db2d2\") " Nov 03 15:35:04.185662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:04.183515 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19da494-26d3-4818-8a32-4af50c7db2d2-maas-api-openshift-ca" pod "f19da494-26d3-4818-8a32-4af50c7db2d2" (UID: "f19da494-26d3-4818-8a32-4af50c7db2d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:35:04.197625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:04.187703 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f19da494-26d3-4818-8a32-4af50c7db2d2-maas-api-xks-ca" pod "f19da494-26d3-4818-8a32-4af50c7db2d2" (UID: "f19da494-26d3-4818-8a32-4af50c7db2d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:35:04.197625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:04.187749 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19da494-26d3-4818-8a32-4af50c7db2d2-kube-api-access-lw74j" pod "f19da494-26d3-4818-8a32-4af50c7db2d2" (UID: "f19da494-26d3-4818-8a32-4af50c7db2d2"). InnerVolumeSpecName "kube-api-access-lw74j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:35:04.284721 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:04.284690 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19da494-26d3-4818-8a32-4af50c7db2d2-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:35:04.284721 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:04.284721 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw74j\" (UniqueName: \"kubernetes.io/projected/f19da494-26d3-4818-8a32-4af50c7db2d2-kube-api-access-lw74j\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:35:04.284927 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:04.284734 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19da494-26d3-4818-8a32-4af50c7db2d2-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:35:04.683446 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:04.683411 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:04.902708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:04.902669 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf" event={"ID":"f19da494-26d3-4818-8a32-4af50c7db2d2","Type":"ContainerDied","Data":"c23a0826a5a4b45973f18dc5a56afacd746c890d42da680f28a3c9f5762ec45b"} Nov 03 15:35:04.902918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:04.902732 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c23a0826a5a4b45973f18dc5a56afacd746c890d42da680f28a3c9f5762ec45b" Nov 03 15:35:04.902918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:04.902690 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf" Nov 03 15:35:06.660016 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:06.659972 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:06.853968 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:06.853904 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:07.054264 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:07.054234 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5"] Nov 03 15:35:07.056437 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:07.056413 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:07.092826 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:07.092791 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab8b4ee-9h8w5"] Nov 03 15:35:08.285994 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:08.285965 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="759a1535-c79d-4518-ad7f-4219c9d28059" path="/var/lib/kubelet/pods/759a1535-c79d-4518-ad7f-4219c9d28059/volumes" Nov 03 15:35:11.732851 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:11.732812 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:12.052259 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:12.052226 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6"] Nov 03 15:35:12.102331 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:12.102301 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a770e14-b6vj6"] Nov 03 15:35:12.286383 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:12.286349 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9a1d3e5-461b-46d5-bdaa-db9e79c802ef" path="/var/lib/kubelet/pods/a9a1d3e5-461b-46d5-bdaa-db9e79c802ef/volumes" Nov 03 15:35:13.821730 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:13.821685 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:14.017574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:14.017522 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:15.210258 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:15.210226 4557 scope.go:122] "RemoveContainer" containerID="0eabd28494280a4207f7e9260052cf2ad254edee05a062d0502cc8a34ecdb2ea" Nov 03 15:35:15.228249 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:15.228218 4557 scope.go:122] "RemoveContainer" containerID="c551cc89aa396a5bb1b45e929e2c8ea5c331dff3b04aa492f14b7b651cafe575" Nov 03 15:35:15.241505 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:15.241479 4557 scope.go:122] "RemoveContainer" containerID="4d051d6c223a769287659c75f2e7be092cc8e9ff8927044001699a5ae7a33726" Nov 03 15:35:15.252612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:15.252560 4557 scope.go:122] "RemoveContainer" containerID="a6bb316625379253546aaf41154e99c3f9393ec909a7316f3b908b31ba9c615c" Nov 03 15:35:15.263132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:15.263108 4557 scope.go:122] "RemoveContainer" containerID="a74f535326a2a9dfb2e7d00d1090274de9e02ac21b7433b1eb8ced37d4bc65a6" Nov 03 15:35:15.275017 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:15.274990 4557 scope.go:122] "RemoveContainer" containerID="f293c81635e6105e8a23601860f6360af71c896a3f532a7d064f4a6912ff0695" Nov 03 15:35:15.286197 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:15.286176 4557 scope.go:122] "RemoveContainer" containerID="9c9a43ff450ba3b75da729d10a05d5fef8bee2af32b30db984ac2168d03ae58f" Nov 03 15:35:15.296751 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:15.296728 4557 scope.go:122] "RemoveContainer" containerID="adfbf2d27191a42b2690a305edf9154e20cf88fce10fa4bcc43b80533856dd19" Nov 03 15:35:16.529825 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:35:16.529790 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:35:16.716001 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:16.715950 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:20.051323 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:20.051288 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl"] Nov 03 15:35:20.101191 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:20.101166 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b85f9f9-zjhbl"] Nov 03 15:35:20.287506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:20.287474 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a8f0988-165b-4bee-85c2-357bfb4e4a60" path="/var/lib/kubelet/pods/5a8f0988-165b-4bee-85c2-357bfb4e4a60/volumes" Nov 03 15:35:21.515111 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:21.515073 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:26.260901 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:26.260865 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:26.454889 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:26.454853 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:26.648183 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:26.648155 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:30.815950 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:30.815910 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:31.335364 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.335331 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst"] Nov 03 15:35:31.335957 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.335939 4557 cpu_manager.go:414] "containerMap: removing container" podUID="f19da494-26d3-4818-8a32-4af50c7db2d2" containerName="revoke-keys" Nov 03 15:35:31.335957 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.335958 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19da494-26d3-4818-8a32-4af50c7db2d2" containerName="revoke-keys" Nov 03 15:35:31.336078 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.336066 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19da494-26d3-4818-8a32-4af50c7db2d2" containerName="revoke-keys" Nov 03 15:35:31.341732 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.341711 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.377920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.376569 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.377920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.376673 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.377920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.376844 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.377920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.376878 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.377920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.376909 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74678262-2f6e-41a6-aa2b-00b64aa2c545-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.377920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.376954 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.377920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.376986 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.377920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.377040 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfvrt\" (UniqueName: \"kubernetes.io/projected/74678262-2f6e-41a6-aa2b-00b64aa2c545-kube-api-access-jfvrt\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.377920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.377075 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.478603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.462129 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:31.480425 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.473064 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5cad072f-openshift-default-dockercfg-2j2g5\"" Nov 03 15:35:31.565478 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.555864 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst"] Nov 03 15:35:31.568274 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.568216 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.568549 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.568518 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.568716 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.568690 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74678262-2f6e-41a6-aa2b-00b64aa2c545-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.568977 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.568947 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.569186 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.569160 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.569393 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.569377 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvrt\" (UniqueName: \"kubernetes.io/projected/74678262-2f6e-41a6-aa2b-00b64aa2c545-kube-api-access-jfvrt\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.569568 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.569541 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.569943 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.569913 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.570264 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.570228 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.571006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.570974 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.571549 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.571504 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.581777 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.581758 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.583245 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.583221 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74678262-2f6e-41a6-aa2b-00b64aa2c545-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.583846 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.583824 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.630196 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.611178 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.630196 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.622196 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.651254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.651221 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.653073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.653045 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfvrt\" (UniqueName: \"kubernetes.io/projected/74678262-2f6e-41a6-aa2b-00b64aa2c545-kube-api-access-jfvrt\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.658602 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.658414 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:31.686710 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:31.686653 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:32.051505 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:32.051469 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk"] Nov 03 15:35:32.055000 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:32.054979 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13702e89-cz2xk"] Nov 03 15:35:32.110805 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:32.110776 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst"] Nov 03 15:35:32.111129 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:35:32.111109 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74678262_2f6e_41a6_aa2b_00b64aa2c545.slice/crio-c3f56b000a0fdf20cb9c89e740412de5865ba5e66da1243eefea75c94eb6b251 WatchSource:0}: Error finding container c3f56b000a0fdf20cb9c89e740412de5865ba5e66da1243eefea75c94eb6b251: Status 404 returned error can't find the container with id c3f56b000a0fdf20cb9c89e740412de5865ba5e66da1243eefea75c94eb6b251 Nov 03 15:35:32.113930 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:32.113902 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:35:32.114000 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:32.113972 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:35:32.114037 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:32.114002 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:35:32.122231 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:32.122205 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" event={"ID":"74678262-2f6e-41a6-aa2b-00b64aa2c545","Type":"ContainerStarted","Data":"c3f56b000a0fdf20cb9c89e740412de5865ba5e66da1243eefea75c94eb6b251"} Nov 03 15:35:32.288214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:32.288179 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e9334a6-331a-4402-b566-0ff28d91dd5e" path="/var/lib/kubelet/pods/3e9334a6-331a-4402-b566-0ff28d91dd5e/volumes" Nov 03 15:35:33.131147 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:33.131106 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" event={"ID":"74678262-2f6e-41a6-aa2b-00b64aa2c545","Type":"ContainerStarted","Data":"8f98e891a9fd39a67d070baca9938c500b6259f4491099f479f7f973c4ca8b32"} Nov 03 15:35:33.306696 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:33.306648 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" podStartSLOduration=2.306633484 podStartE2EDuration="2.306633484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:35:33.303056287 +0000 UTC m=+1643.146177857" watchObservedRunningTime="2026-11-03 15:35:33.306633484 +0000 UTC m=+1643.149755096" Nov 03 15:35:33.659661 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:33.659620 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:33.665041 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:33.665014 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:34.138689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:34.138657 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:34.139488 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:34.139472 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:35:36.183425 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:36.183395 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:36.416003 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:36.415967 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:36.625632 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:36.622415 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:39.388175 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:39.388142 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:39.592636 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:39.590744 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:41.044911 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:41.044873 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:41.608474 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:41.608425 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:45.852007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:45.851964 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:46.532176 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:35:46.532138 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:35:47.051929 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:47.051896 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb"] Nov 03 15:35:47.102274 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:47.102237 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189933f2-cbzvb"] Nov 03 15:35:47.105782 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:47.105754 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh"] Nov 03 15:35:47.109524 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:47.109504 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05dca9f-j47rh"] Nov 03 15:35:47.437883 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:47.437810 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:47.643801 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:47.643762 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:47.831958 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:47.831924 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:47.909951 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:47.909923 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf"] Nov 03 15:35:47.944719 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:47.944691 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dfec2d6c-gt8zf"] Nov 03 15:35:48.102031 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:48.101940 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:48.303381 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:48.303349 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="442544d0-79c6-486e-a3bd-163752dc0112" path="/var/lib/kubelet/pods/442544d0-79c6-486e-a3bd-163752dc0112/volumes" Nov 03 15:35:48.303900 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:48.303885 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f472ac9-570a-4124-ba65-e9f81b49342d" path="/var/lib/kubelet/pods/4f472ac9-570a-4124-ba65-e9f81b49342d/volumes" Nov 03 15:35:48.304356 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:48.304343 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19da494-26d3-4818-8a32-4af50c7db2d2" path="/var/lib/kubelet/pods/f19da494-26d3-4818-8a32-4af50c7db2d2/volumes" Nov 03 15:35:48.888971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:48.888916 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:50.080210 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:50.080170 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:50.357028 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:50.356937 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:52.961516 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:52.961481 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:54.656359 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:54.656329 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:54.847796 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:54.847768 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:57.964415 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:57.964374 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:58.056447 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:58.056395 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv"] Nov 03 15:35:58.109154 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:58.109117 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r"] Nov 03 15:35:58.114971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:58.114943 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea6ae55-rczwv"] Nov 03 15:35:58.118952 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:58.118927 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f2d8de3-x288r"] Nov 03 15:35:58.170252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:58.170210 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:35:58.287553 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:58.287448 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2bfb1a1-688c-4056-9195-6f08a265e92c" path="/var/lib/kubelet/pods/d2bfb1a1-688c-4056-9195-6f08a265e92c/volumes" Nov 03 15:35:58.288126 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:58.288101 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e488d5c5-a263-4aab-9bde-705df88b1bd8" path="/var/lib/kubelet/pods/e488d5c5-a263-4aab-9bde-705df88b1bd8/volumes" Nov 03 15:35:58.371223 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:35:58.371178 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:01.050174 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:01.050138 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6"] Nov 03 15:36:01.099360 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:01.099328 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dbb6da8-2kvg6"] Nov 03 15:36:02.286753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:02.286707 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81c3afc1-1565-4482-8be4-dc95791b8426" path="/var/lib/kubelet/pods/81c3afc1-1565-4482-8be4-dc95791b8426/volumes" Nov 03 15:36:03.209007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.208971 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s"] Nov 03 15:36:03.212935 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.212893 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.266245 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.266218 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-42595854-openshift-default-dockercfg-85gc8\"" Nov 03 15:36:03.319949 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.319918 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s"] Nov 03 15:36:03.392141 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.392102 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-data\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.392326 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.392231 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istiod-ca-cert\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.392326 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.392268 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-credential-socket\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.392326 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.392289 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-workload-certs\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.392497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.392381 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-token\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.392564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.392498 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cvdw\" (UniqueName: \"kubernetes.io/projected/efec6fa1-66c0-4013-b3d0-9d08a04d8830-kube-api-access-4cvdw\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.392564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.392542 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-envoy\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.392703 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.392654 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-workload-socket\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.392825 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.392803 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-podinfo\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.494123 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.494034 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-data\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.494304 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.494134 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istiod-ca-cert\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.494304 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.494172 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-credential-socket\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.494304 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.494197 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-workload-certs\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.494304 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.494226 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-token\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.494304 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.494275 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cvdw\" (UniqueName: \"kubernetes.io/projected/efec6fa1-66c0-4013-b3d0-9d08a04d8830-kube-api-access-4cvdw\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.494556 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.494313 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-envoy\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.494556 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.494380 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-workload-socket\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.494556 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.494423 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-podinfo\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.494556 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.494498 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-data\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.494801 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.494567 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-credential-socket\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.494801 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.494629 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-workload-certs\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.494914 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.494826 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istiod-ca-cert\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.494914 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.494857 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-workload-socket\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.496745 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.496723 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-envoy\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.496997 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.496979 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-podinfo\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.550070 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.550022 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:03.559012 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.556482 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cvdw\" (UniqueName: \"kubernetes.io/projected/efec6fa1-66c0-4013-b3d0-9d08a04d8830-kube-api-access-4cvdw\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.559012 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.558396 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-token\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-d456s\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:03.827414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:03.827366 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:04.254169 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:04.254094 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:04.265229 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:04.265196 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s"] Nov 03 15:36:04.265755 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:36:04.265723 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefec6fa1_66c0_4013_b3d0_9d08a04d8830.slice/crio-7f4c5be6a236133aa6158743ea39f467f4682e6f79ef0c73d6420c08279d9e34 WatchSource:0}: Error finding container 7f4c5be6a236133aa6158743ea39f467f4682e6f79ef0c73d6420c08279d9e34: Status 404 returned error can't find the container with id 7f4c5be6a236133aa6158743ea39f467f4682e6f79ef0c73d6420c08279d9e34 Nov 03 15:36:04.268926 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:04.268896 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:36:04.269036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:04.268960 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:36:04.269036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:04.268987 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:36:04.381012 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:04.380978 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" event={"ID":"efec6fa1-66c0-4013-b3d0-9d08a04d8830","Type":"ContainerStarted","Data":"a3db4c1589f463eddd7a83823a2914ddfba6a7c531a87d6f83852d1d20d00e13"} Nov 03 15:36:04.381446 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:04.381032 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" event={"ID":"efec6fa1-66c0-4013-b3d0-9d08a04d8830","Type":"ContainerStarted","Data":"7f4c5be6a236133aa6158743ea39f467f4682e6f79ef0c73d6420c08279d9e34"} Nov 03 15:36:04.507741 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:04.507629 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" podStartSLOduration=1.5076084349999999 podStartE2EDuration="1.507608435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:36:04.505392936 +0000 UTC m=+1674.348514479" watchObservedRunningTime="2026-11-03 15:36:04.507608435 +0000 UTC m=+1674.350729977" Nov 03 15:36:04.828241 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:04.828205 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:05.051506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:05.051474 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:05.275016 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:05.274931 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:05.503669 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:05.503631 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:05.716437 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:05.716404 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:05.830063 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:05.830028 4557 patch_prober.go:29] interesting pod/e2e-mt-42595854-openshift-default-8476dc6c85-d456s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 03 15:36:05.830238 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:05.830081 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" podUID="efec6fa1-66c0-4013-b3d0-9d08a04d8830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 03 15:36:05.972643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:05.972540 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:06.827908 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:06.827875 4557 patch_prober.go:29] interesting pod/e2e-mt-42595854-openshift-default-8476dc6c85-d456s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 03 15:36:06.828307 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:06.827932 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" podUID="efec6fa1-66c0-4013-b3d0-9d08a04d8830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 03 15:36:07.827970 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:07.827933 4557 patch_prober.go:29] interesting pod/e2e-mt-42595854-openshift-default-8476dc6c85-d456s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 03 15:36:07.828364 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:07.827990 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" podUID="efec6fa1-66c0-4013-b3d0-9d08a04d8830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 03 15:36:08.828458 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:08.828423 4557 patch_prober.go:29] interesting pod/e2e-mt-42595854-openshift-default-8476dc6c85-d456s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 03 15:36:08.828979 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:08.828494 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" podUID="efec6fa1-66c0-4013-b3d0-9d08a04d8830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 03 15:36:08.844349 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:08.844320 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:09.827656 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:09.827622 4557 patch_prober.go:29] interesting pod/e2e-mt-42595854-openshift-default-8476dc6c85-d456s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 03 15:36:09.827849 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:09.827685 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" podUID="efec6fa1-66c0-4013-b3d0-9d08a04d8830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 03 15:36:10.833036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:10.833006 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:10.833514 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:10.833405 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:10.834180 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:10.834161 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:36:12.913975 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:12.913943 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:13.078568 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:13.078533 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt"] Nov 03 15:36:13.128346 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:13.128313 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35003dd-bkfkt"] Nov 03 15:36:14.213132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:14.213054 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:14.289848 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:14.289805 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d6f0c6b-9501-4e78-9080-79127bf92470" path="/var/lib/kubelet/pods/1d6f0c6b-9501-4e78-9080-79127bf92470/volumes" Nov 03 15:36:15.378001 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:15.377968 4557 scope.go:122] "RemoveContainer" containerID="afe015b317288611d8a871d95a1144d082f4e633f78367ab975168c89d58a9b7" Nov 03 15:36:15.388413 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:15.388395 4557 scope.go:122] "RemoveContainer" containerID="a97fbed3450487774d9e9115b3f40c2690362bf137a10095e370f62588aa7fa5" Nov 03 15:36:15.397480 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:15.397459 4557 scope.go:122] "RemoveContainer" containerID="edc5beccb4be7b7404817ca859fc846301f037bf8d259584d97cf04d16b0d7f4" Nov 03 15:36:15.406125 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:15.406107 4557 scope.go:122] "RemoveContainer" containerID="1b71d64c5183e2f5f7a0bf2bd66cee8b34c40a0f7d19f00da4e32d5b53a5779a" Nov 03 15:36:15.415081 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:15.415050 4557 scope.go:122] "RemoveContainer" containerID="f422b883d10b7dcc6cdc11d84cb368316dbae43530890bc46040e7e42188b815" Nov 03 15:36:15.423683 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:15.423666 4557 scope.go:122] "RemoveContainer" containerID="eea172ff772d874ee99a3bf8cd3399b3061a638ed9e06991ce3cea48edfa1c77" Nov 03 15:36:15.432188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:15.432160 4557 scope.go:122] "RemoveContainer" containerID="66ae51747982d275392d6d7a682bb35e72a83742edc1e86b1ef661b7621532d9" Nov 03 15:36:15.440327 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:15.440306 4557 scope.go:122] "RemoveContainer" containerID="945290b9996c9ec6b16b50389149b43f24c57f76209b71f2a5eb602feacca3cc" Nov 03 15:36:15.448779 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:15.448761 4557 scope.go:122] "RemoveContainer" containerID="396d5213d238de9a2ee78c3c96c0848449dd74c2941aaa68800cb48645bf1267" Nov 03 15:36:15.456729 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:15.456710 4557 scope.go:122] "RemoveContainer" containerID="73c294e7aefa6bc62ec893ad3f1f8bf1b9ebc1a74ed26a4a79d561c2632eb6c2" Nov 03 15:36:16.530527 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:36:16.530493 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:36:19.030873 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:19.030832 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:24.085002 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:24.084966 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:26.801263 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:26.801224 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd"] Nov 03 15:36:26.806849 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:26.806829 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd" Nov 03 15:36:26.864081 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:26.864048 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd"] Nov 03 15:36:26.870208 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:26.870167 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7596d6db-b29f-4686-ada3-d58864dcc8c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14ae485a-kfntd\" (UID: \"7596d6db-b29f-4686-ada3-d58864dcc8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd" Nov 03 15:36:26.870433 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:26.870383 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7596d6db-b29f-4686-ada3-d58864dcc8c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14ae485a-kfntd\" (UID: \"7596d6db-b29f-4686-ada3-d58864dcc8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd" Nov 03 15:36:26.870530 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:26.870493 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvrt9\" (UniqueName: \"kubernetes.io/projected/7596d6db-b29f-4686-ada3-d58864dcc8c1-kube-api-access-nvrt9\") pod \"maas-api-revoke-sub-14ae485a-kfntd\" (UID: \"7596d6db-b29f-4686-ada3-d58864dcc8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd" Nov 03 15:36:26.909619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:26.909188 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:36:26.909619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:26.909235 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:36:26.971815 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:26.971772 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrt9\" (UniqueName: \"kubernetes.io/projected/7596d6db-b29f-4686-ada3-d58864dcc8c1-kube-api-access-nvrt9\") pod \"maas-api-revoke-sub-14ae485a-kfntd\" (UID: \"7596d6db-b29f-4686-ada3-d58864dcc8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd" Nov 03 15:36:26.972013 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:26.971875 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7596d6db-b29f-4686-ada3-d58864dcc8c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14ae485a-kfntd\" (UID: \"7596d6db-b29f-4686-ada3-d58864dcc8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd" Nov 03 15:36:26.972087 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:26.972040 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7596d6db-b29f-4686-ada3-d58864dcc8c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14ae485a-kfntd\" (UID: \"7596d6db-b29f-4686-ada3-d58864dcc8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd" Nov 03 15:36:26.972713 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:26.972687 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7596d6db-b29f-4686-ada3-d58864dcc8c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14ae485a-kfntd\" (UID: \"7596d6db-b29f-4686-ada3-d58864dcc8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd" Nov 03 15:36:26.974181 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:26.974159 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7596d6db-b29f-4686-ada3-d58864dcc8c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14ae485a-kfntd\" (UID: \"7596d6db-b29f-4686-ada3-d58864dcc8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd" Nov 03 15:36:27.033243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:27.033208 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvrt9\" (UniqueName: \"kubernetes.io/projected/7596d6db-b29f-4686-ada3-d58864dcc8c1-kube-api-access-nvrt9\") pod \"maas-api-revoke-sub-14ae485a-kfntd\" (UID: \"7596d6db-b29f-4686-ada3-d58864dcc8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd" Nov 03 15:36:27.121090 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:27.121011 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd" Nov 03 15:36:27.503011 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:36:27.502977 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7596d6db_b29f_4686_ada3_d58864dcc8c1.slice/crio-ce5c911ff3de596e9d554ee9ea949581dbe586b80b11e1e3571c0ba3a242699a WatchSource:0}: Error finding container ce5c911ff3de596e9d554ee9ea949581dbe586b80b11e1e3571c0ba3a242699a: Status 404 returned error can't find the container with id ce5c911ff3de596e9d554ee9ea949581dbe586b80b11e1e3571c0ba3a242699a Nov 03 15:36:27.503491 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:27.503462 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd"] Nov 03 15:36:27.582399 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:27.582367 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd" event={"ID":"7596d6db-b29f-4686-ada3-d58864dcc8c1","Type":"ContainerStarted","Data":"ce5c911ff3de596e9d554ee9ea949581dbe586b80b11e1e3571c0ba3a242699a"} Nov 03 15:36:28.591033 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:28.590997 4557 generic.go:356] "Generic (PLEG): container finished" podID="7596d6db-b29f-4686-ada3-d58864dcc8c1" containerID="9ebe4eb678040c576f1636f877a843c44368b708b9f046e25da0ed2f9ca4ae53" exitCode=0 Nov 03 15:36:28.591441 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:28.591083 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd" event={"ID":"7596d6db-b29f-4686-ada3-d58864dcc8c1","Type":"ContainerDied","Data":"9ebe4eb678040c576f1636f877a843c44368b708b9f046e25da0ed2f9ca4ae53"} Nov 03 15:36:28.823370 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:28.823324 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:29.237733 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:29.237640 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:29.442216 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:29.441543 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:29.775553 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:29.775450 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd" Nov 03 15:36:29.817234 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:29.817202 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7596d6db-b29f-4686-ada3-d58864dcc8c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7596d6db-b29f-4686-ada3-d58864dcc8c1-maas-api-xks-ca\") pod \"7596d6db-b29f-4686-ada3-d58864dcc8c1\" (UID: \"7596d6db-b29f-4686-ada3-d58864dcc8c1\") " Nov 03 15:36:29.817439 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:29.817282 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7596d6db-b29f-4686-ada3-d58864dcc8c1-kube-api-access-nvrt9\" (UniqueName: \"kubernetes.io/projected/7596d6db-b29f-4686-ada3-d58864dcc8c1-kube-api-access-nvrt9\") pod \"7596d6db-b29f-4686-ada3-d58864dcc8c1\" (UID: \"7596d6db-b29f-4686-ada3-d58864dcc8c1\") " Nov 03 15:36:29.817439 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:29.817426 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7596d6db-b29f-4686-ada3-d58864dcc8c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7596d6db-b29f-4686-ada3-d58864dcc8c1-maas-api-openshift-ca\") pod \"7596d6db-b29f-4686-ada3-d58864dcc8c1\" (UID: \"7596d6db-b29f-4686-ada3-d58864dcc8c1\") " Nov 03 15:36:29.818092 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:29.817988 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7596d6db-b29f-4686-ada3-d58864dcc8c1-maas-api-openshift-ca" pod "7596d6db-b29f-4686-ada3-d58864dcc8c1" (UID: "7596d6db-b29f-4686-ada3-d58864dcc8c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:36:29.820881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:29.820855 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7596d6db-b29f-4686-ada3-d58864dcc8c1-kube-api-access-nvrt9" pod "7596d6db-b29f-4686-ada3-d58864dcc8c1" (UID: "7596d6db-b29f-4686-ada3-d58864dcc8c1"). InnerVolumeSpecName "kube-api-access-nvrt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:36:29.822708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:29.822684 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7596d6db-b29f-4686-ada3-d58864dcc8c1-maas-api-xks-ca" pod "7596d6db-b29f-4686-ada3-d58864dcc8c1" (UID: "7596d6db-b29f-4686-ada3-d58864dcc8c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:36:29.919288 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:29.919208 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7596d6db-b29f-4686-ada3-d58864dcc8c1-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:29.919288 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:29.919241 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7596d6db-b29f-4686-ada3-d58864dcc8c1-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:29.919288 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:29.919255 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvrt9\" (UniqueName: \"kubernetes.io/projected/7596d6db-b29f-4686-ada3-d58864dcc8c1-kube-api-access-nvrt9\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:30.096683 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:30.090676 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:30.609524 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:30.609445 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd" Nov 03 15:36:30.609524 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:30.609494 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd" event={"ID":"7596d6db-b29f-4686-ada3-d58864dcc8c1","Type":"ContainerDied","Data":"ce5c911ff3de596e9d554ee9ea949581dbe586b80b11e1e3571c0ba3a242699a"} Nov 03 15:36:30.609788 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:30.609555 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce5c911ff3de596e9d554ee9ea949581dbe586b80b11e1e3571c0ba3a242699a" Nov 03 15:36:30.612016 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:30.611994 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:34.642119 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:34.642062 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:35.167616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:35.167549 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:35.345763 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:35.345727 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8"] Nov 03 15:36:35.345967 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:35.345940 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" podUID="53cb6efb-85fe-4c12-962c-436a99fedfa5" containerName="istio-proxy" containerID="cri-o://711e472abcac085b515eeb310992e7697a4a08b1e5c7157acb66f3d500de8703" gracePeriod=30 Nov 03 15:36:35.376522 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:35.376489 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:35.394944 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:35.394906 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx"] Nov 03 15:36:35.395176 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:35.395150 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" podUID="a589a74b-03fc-4a08-bd68-9ad4066f7ce1" containerName="istio-proxy" containerID="cri-o://eb2aa0669f07bc636d3e348d49eb3be5adcb85fbabd84abcf306496776996860" gracePeriod=30 Nov 03 15:36:35.398621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:35.398594 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb"] Nov 03 15:36:35.398813 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:35.398793 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" podUID="e362b475-7544-4095-a465-24905a71c6c0" containerName="istio-proxy" containerID="cri-o://9387861b5b3928ba16c1e26e80a13ef6941619ab5f09fcf231a13bb8bbc39874" gracePeriod=30 Nov 03 15:36:35.444381 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:35.444311 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z"] Nov 03 15:36:35.444525 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:35.444504 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" podUID="3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" containerName="istio-proxy" containerID="cri-o://dc73687addedf98708f783dfaafb06080d641677899ec127943fc2b696cb2d88" gracePeriod=30 Nov 03 15:36:35.596319 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:35.596236 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:37.838056 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:37.837990 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:38.437867 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:38.437800 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:38.630707 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:38.630668 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:40.126354 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.126314 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:40.300038 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.299999 4557 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:36:40.300180 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.300066 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" podUID="53cb6efb-85fe-4c12-962c-436a99fedfa5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:36:40.713526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.713440 4557 generic.go:356] "Generic (PLEG): container finished" podID="3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" containerID="dc73687addedf98708f783dfaafb06080d641677899ec127943fc2b696cb2d88" exitCode=0 Nov 03 15:36:40.713717 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.713620 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" event={"ID":"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c","Type":"ContainerDied","Data":"dc73687addedf98708f783dfaafb06080d641677899ec127943fc2b696cb2d88"} Nov 03 15:36:40.715320 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.715294 4557 generic.go:356] "Generic (PLEG): container finished" podID="a589a74b-03fc-4a08-bd68-9ad4066f7ce1" containerID="eb2aa0669f07bc636d3e348d49eb3be5adcb85fbabd84abcf306496776996860" exitCode=0 Nov 03 15:36:40.715567 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.715469 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" event={"ID":"a589a74b-03fc-4a08-bd68-9ad4066f7ce1","Type":"ContainerDied","Data":"eb2aa0669f07bc636d3e348d49eb3be5adcb85fbabd84abcf306496776996860"} Nov 03 15:36:40.715823 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.715800 4557 scope.go:122] "RemoveContainer" containerID="8171fe2f20c451c3c9918071e458960fac162f4fe9f0a0259199fc7f2bf88e8e" Nov 03 15:36:40.719018 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.718949 4557 generic.go:356] "Generic (PLEG): container finished" podID="e362b475-7544-4095-a465-24905a71c6c0" containerID="9387861b5b3928ba16c1e26e80a13ef6941619ab5f09fcf231a13bb8bbc39874" exitCode=0 Nov 03 15:36:40.719196 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.719173 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" event={"ID":"e362b475-7544-4095-a465-24905a71c6c0","Type":"ContainerDied","Data":"9387861b5b3928ba16c1e26e80a13ef6941619ab5f09fcf231a13bb8bbc39874"} Nov 03 15:36:40.721946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.721905 4557 generic.go:356] "Generic (PLEG): container finished" podID="53cb6efb-85fe-4c12-962c-436a99fedfa5" containerID="711e472abcac085b515eeb310992e7697a4a08b1e5c7157acb66f3d500de8703" exitCode=0 Nov 03 15:36:40.722016 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.721962 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" event={"ID":"53cb6efb-85fe-4c12-962c-436a99fedfa5","Type":"ContainerDied","Data":"711e472abcac085b515eeb310992e7697a4a08b1e5c7157acb66f3d500de8703"} Nov 03 15:36:40.797465 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.797430 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:36:40.863852 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.863041 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-credential-socket\") pod \"53cb6efb-85fe-4c12-962c-436a99fedfa5\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " Nov 03 15:36:40.865285 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.865252 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-podinfo\") pod \"53cb6efb-85fe-4c12-962c-436a99fedfa5\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " Nov 03 15:36:40.870576 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.870541 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53cb6efb-85fe-4c12-962c-436a99fedfa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53cb6efb-85fe-4c12-962c-436a99fedfa5-istiod-ca-cert\") pod \"53cb6efb-85fe-4c12-962c-436a99fedfa5\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " Nov 03 15:36:40.870738 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.870691 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-envoy\") pod \"53cb6efb-85fe-4c12-962c-436a99fedfa5\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " Nov 03 15:36:40.870738 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.870731 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-workload-certs\") pod \"53cb6efb-85fe-4c12-962c-436a99fedfa5\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " Nov 03 15:36:40.870881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.870808 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-data\") pod \"53cb6efb-85fe-4c12-962c-436a99fedfa5\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " Nov 03 15:36:40.870881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.870863 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-workload-socket\") pod \"53cb6efb-85fe-4c12-962c-436a99fedfa5\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " Nov 03 15:36:40.871003 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.870903 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-token\" (UniqueName: \"kubernetes.io/projected/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-token\") pod \"53cb6efb-85fe-4c12-962c-436a99fedfa5\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " Nov 03 15:36:40.871003 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.870934 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53cb6efb-85fe-4c12-962c-436a99fedfa5-kube-api-access-qbvl6\" (UniqueName: \"kubernetes.io/projected/53cb6efb-85fe-4c12-962c-436a99fedfa5-kube-api-access-qbvl6\") pod \"53cb6efb-85fe-4c12-962c-436a99fedfa5\" (UID: \"53cb6efb-85fe-4c12-962c-436a99fedfa5\") " Nov 03 15:36:40.872678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.863342 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-credential-socket" pod "53cb6efb-85fe-4c12-962c-436a99fedfa5" (UID: "53cb6efb-85fe-4c12-962c-436a99fedfa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:40.872678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.872145 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-data" pod "53cb6efb-85fe-4c12-962c-436a99fedfa5" (UID: "53cb6efb-85fe-4c12-962c-436a99fedfa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:40.872678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.872340 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-workload-socket" pod "53cb6efb-85fe-4c12-962c-436a99fedfa5" (UID: "53cb6efb-85fe-4c12-962c-436a99fedfa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:40.876260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.876065 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-token" pod "53cb6efb-85fe-4c12-962c-436a99fedfa5" (UID: "53cb6efb-85fe-4c12-962c-436a99fedfa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:36:40.876519 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.876487 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53cb6efb-85fe-4c12-962c-436a99fedfa5-istiod-ca-cert" pod "53cb6efb-85fe-4c12-962c-436a99fedfa5" (UID: "53cb6efb-85fe-4c12-962c-436a99fedfa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:36:40.889599 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.889526 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-workload-certs" pod "53cb6efb-85fe-4c12-962c-436a99fedfa5" (UID: "53cb6efb-85fe-4c12-962c-436a99fedfa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:40.903881 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.903850 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53cb6efb-85fe-4c12-962c-436a99fedfa5-kube-api-access-qbvl6" pod "53cb6efb-85fe-4c12-962c-436a99fedfa5" (UID: "53cb6efb-85fe-4c12-962c-436a99fedfa5"). InnerVolumeSpecName "kube-api-access-qbvl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:36:40.904342 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.904296 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-podinfo" pod "53cb6efb-85fe-4c12-962c-436a99fedfa5" (UID: "53cb6efb-85fe-4c12-962c-436a99fedfa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:36:40.904342 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.904305 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-envoy" pod "53cb6efb-85fe-4c12-962c-436a99fedfa5" (UID: "53cb6efb-85fe-4c12-962c-436a99fedfa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:40.915335 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.915313 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:36:40.963268 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.963228 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:36:40.974079 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.974012 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.974079 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.974062 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.974267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.974081 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.974267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.974096 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbvl6\" (UniqueName: \"kubernetes.io/projected/53cb6efb-85fe-4c12-962c-436a99fedfa5-kube-api-access-qbvl6\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.974267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.974132 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.974267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.974148 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.974267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.974162 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53cb6efb-85fe-4c12-962c-436a99fedfa5-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.974267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.974178 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.974267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.974209 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53cb6efb-85fe-4c12-962c-436a99fedfa5-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:40.975371 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:40.975235 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:36:41.074937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.074909 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-podinfo\") pod \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " Nov 03 15:36:41.075128 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.074955 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e362b475-7544-4095-a465-24905a71c6c0-kube-api-access-qczms\" (UniqueName: \"kubernetes.io/projected/e362b475-7544-4095-a465-24905a71c6c0-kube-api-access-qczms\") pod \"e362b475-7544-4095-a465-24905a71c6c0\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " Nov 03 15:36:41.075128 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.074990 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-credential-socket\") pod \"e362b475-7544-4095-a465-24905a71c6c0\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " Nov 03 15:36:41.075283 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075152 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-token\" (UniqueName: \"kubernetes.io/projected/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-token\") pod \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " Nov 03 15:36:41.075283 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075211 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-credential-socket" pod "e362b475-7544-4095-a465-24905a71c6c0" (UID: "e362b475-7544-4095-a465-24905a71c6c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:41.075283 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075223 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e362b475-7544-4095-a465-24905a71c6c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e362b475-7544-4095-a465-24905a71c6c0-istiod-ca-cert\") pod \"e362b475-7544-4095-a465-24905a71c6c0\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " Nov 03 15:36:41.075443 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075316 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-credential-socket\") pod \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " Nov 03 15:36:41.075443 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075351 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-istio-data\") pod \"e362b475-7544-4095-a465-24905a71c6c0\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " Nov 03 15:36:41.075443 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075383 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-kube-api-access-pwkml\" (UniqueName: \"kubernetes.io/projected/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-kube-api-access-pwkml\") pod \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " Nov 03 15:36:41.075610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075476 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-envoy\") pod \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " Nov 03 15:36:41.075610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075536 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-workload-socket\") pod \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " Nov 03 15:36:41.075610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075563 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-credential-socket" pod "3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" (UID: "3a8227fd-da10-4bd8-b86f-5f7fc48ef40c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:41.075771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075614 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-istio-data" pod "e362b475-7544-4095-a465-24905a71c6c0" (UID: "e362b475-7544-4095-a465-24905a71c6c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:41.075771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075656 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-workload-socket\") pod \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " Nov 03 15:36:41.075771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075657 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e362b475-7544-4095-a465-24905a71c6c0-istiod-ca-cert" pod "e362b475-7544-4095-a465-24905a71c6c0" (UID: "e362b475-7544-4095-a465-24905a71c6c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:36:41.075771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075715 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-workload-certs\") pod \"e362b475-7544-4095-a465-24905a71c6c0\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " Nov 03 15:36:41.075771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075730 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-workload-socket" pod "3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" (UID: "3a8227fd-da10-4bd8-b86f-5f7fc48ef40c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:41.075771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075748 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-workload-certs\") pod \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " Nov 03 15:36:41.076104 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075822 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e362b475-7544-4095-a465-24905a71c6c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e362b475-7544-4095-a465-24905a71c6c0-istio-podinfo\") pod \"e362b475-7544-4095-a465-24905a71c6c0\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " Nov 03 15:36:41.076104 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075878 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istiod-ca-cert\") pod \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " Nov 03 15:36:41.076104 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075923 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-envoy\") pod \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " Nov 03 15:36:41.076104 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075928 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-workload-certs" pod "e362b475-7544-4095-a465-24905a71c6c0" (UID: "e362b475-7544-4095-a465-24905a71c6c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:41.076104 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075970 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e362b475-7544-4095-a465-24905a71c6c0-istio-token\" (UniqueName: \"kubernetes.io/projected/e362b475-7544-4095-a465-24905a71c6c0-istio-token\") pod \"e362b475-7544-4095-a465-24905a71c6c0\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " Nov 03 15:36:41.076104 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.075976 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-workload-socket" pod "a589a74b-03fc-4a08-bd68-9ad4066f7ce1" (UID: "a589a74b-03fc-4a08-bd68-9ad4066f7ce1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:41.076104 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.076010 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-podinfo\") pod \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " Nov 03 15:36:41.076104 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.076033 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-workload-certs" pod "3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" (UID: "3a8227fd-da10-4bd8-b86f-5f7fc48ef40c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:41.076104 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.076055 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-kube-api-access-tv7p2\" (UniqueName: \"kubernetes.io/projected/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-kube-api-access-tv7p2\") pod \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " Nov 03 15:36:41.076569 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.076127 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-credential-socket\") pod \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " Nov 03 15:36:41.076569 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.076166 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-workload-socket\") pod \"e362b475-7544-4095-a465-24905a71c6c0\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " Nov 03 15:36:41.076569 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.076196 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-data\") pod \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\" (UID: \"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c\") " Nov 03 15:36:41.076569 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.076221 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istiod-ca-cert" pod "3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" (UID: "3a8227fd-da10-4bd8-b86f-5f7fc48ef40c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:36:41.076569 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.076278 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-workload-certs\") pod \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " Nov 03 15:36:41.076569 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.076338 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-token\" (UniqueName: \"kubernetes.io/projected/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-token\") pod \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " Nov 03 15:36:41.076569 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.076370 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istiod-ca-cert\") pod \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " Nov 03 15:36:41.076569 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.076407 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-istio-envoy\") pod \"e362b475-7544-4095-a465-24905a71c6c0\" (UID: \"e362b475-7544-4095-a465-24905a71c6c0\") " Nov 03 15:36:41.076569 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.076497 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-data\") pod \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\" (UID: \"a589a74b-03fc-4a08-bd68-9ad4066f7ce1\") " Nov 03 15:36:41.077096 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.076626 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-workload-socket" pod "e362b475-7544-4095-a465-24905a71c6c0" (UID: "e362b475-7544-4095-a465-24905a71c6c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:41.077096 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.076726 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-workload-certs" pod "a589a74b-03fc-4a08-bd68-9ad4066f7ce1" (UID: "a589a74b-03fc-4a08-bd68-9ad4066f7ce1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:41.077096 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.076837 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-credential-socket" pod "a589a74b-03fc-4a08-bd68-9ad4066f7ce1" (UID: "a589a74b-03fc-4a08-bd68-9ad4066f7ce1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:41.077096 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.076934 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-data" pod "a589a74b-03fc-4a08-bd68-9ad4066f7ce1" (UID: "a589a74b-03fc-4a08-bd68-9ad4066f7ce1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:41.086974 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.077332 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istiod-ca-cert" pod "a589a74b-03fc-4a08-bd68-9ad4066f7ce1" (UID: "a589a74b-03fc-4a08-bd68-9ad4066f7ce1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:36:41.086974 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.076978 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-data" pod "3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" (UID: "3a8227fd-da10-4bd8-b86f-5f7fc48ef40c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:41.086974 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.086726 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-token" pod "3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" (UID: "3a8227fd-da10-4bd8-b86f-5f7fc48ef40c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:36:41.088056 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.088025 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-envoy" pod "3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" (UID: "3a8227fd-da10-4bd8-b86f-5f7fc48ef40c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:41.089094 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089050 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-podinfo" pod "a589a74b-03fc-4a08-bd68-9ad4066f7ce1" (UID: "a589a74b-03fc-4a08-bd68-9ad4066f7ce1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:36:41.089430 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089401 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e362b475-7544-4095-a465-24905a71c6c0-istio-token" pod "e362b475-7544-4095-a465-24905a71c6c0" (UID: "e362b475-7544-4095-a465-24905a71c6c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:36:41.089675 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089653 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.089766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089680 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e362b475-7544-4095-a465-24905a71c6c0-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.089766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089701 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.089766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089716 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.089766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089728 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.089766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089742 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.089766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089756 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.089766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089769 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.090406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089781 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.090406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089720 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-kube-api-access-pwkml" pod "a589a74b-03fc-4a08-bd68-9ad4066f7ce1" (UID: "a589a74b-03fc-4a08-bd68-9ad4066f7ce1"). InnerVolumeSpecName "kube-api-access-pwkml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:36:41.090406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089793 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.090406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089769 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-istio-envoy" pod "e362b475-7544-4095-a465-24905a71c6c0" (UID: "e362b475-7544-4095-a465-24905a71c6c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:41.090406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089807 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.090406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089820 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e362b475-7544-4095-a465-24905a71c6c0-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.090406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089836 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.090406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089848 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.090406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089859 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.090406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089873 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.090406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089885 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.090406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089900 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.090406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.089913 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.090406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.090178 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-podinfo" pod "3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" (UID: "3a8227fd-da10-4bd8-b86f-5f7fc48ef40c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:36:41.092164 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.091778 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e362b475-7544-4095-a465-24905a71c6c0-istio-podinfo" pod "e362b475-7544-4095-a465-24905a71c6c0" (UID: "e362b475-7544-4095-a465-24905a71c6c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:36:41.092224 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.092198 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-kube-api-access-tv7p2" pod "3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" (UID: "3a8227fd-da10-4bd8-b86f-5f7fc48ef40c"). InnerVolumeSpecName "kube-api-access-tv7p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:36:41.092390 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.092333 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-envoy" pod "a589a74b-03fc-4a08-bd68-9ad4066f7ce1" (UID: "a589a74b-03fc-4a08-bd68-9ad4066f7ce1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:36:41.092937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.092912 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e362b475-7544-4095-a465-24905a71c6c0-kube-api-access-qczms" pod "e362b475-7544-4095-a465-24905a71c6c0" (UID: "e362b475-7544-4095-a465-24905a71c6c0"). InnerVolumeSpecName "kube-api-access-qczms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:36:41.093098 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.093069 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-token" pod "a589a74b-03fc-4a08-bd68-9ad4066f7ce1" (UID: "a589a74b-03fc-4a08-bd68-9ad4066f7ce1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:36:41.191298 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.191263 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e362b475-7544-4095-a465-24905a71c6c0-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.191298 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.191291 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.191298 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.191301 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.191768 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.191312 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv7p2\" (UniqueName: \"kubernetes.io/projected/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c-kube-api-access-tv7p2\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.191768 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.191326 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.191768 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.191339 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e362b475-7544-4095-a465-24905a71c6c0-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.191768 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.191348 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qczms\" (UniqueName: \"kubernetes.io/projected/e362b475-7544-4095-a465-24905a71c6c0-kube-api-access-qczms\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.191768 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.191359 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwkml\" (UniqueName: \"kubernetes.io/projected/a589a74b-03fc-4a08-bd68-9ad4066f7ce1-kube-api-access-pwkml\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:36:41.730844 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.730752 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" Nov 03 15:36:41.730844 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.730781 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8" event={"ID":"53cb6efb-85fe-4c12-962c-436a99fedfa5","Type":"ContainerDied","Data":"b26def6448f397781d37d11338e4dcc1deec225bc433e6ba249f93f70792b13b"} Nov 03 15:36:41.731066 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.730849 4557 scope.go:122] "RemoveContainer" containerID="711e472abcac085b515eeb310992e7697a4a08b1e5c7157acb66f3d500de8703" Nov 03 15:36:41.732410 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.732392 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" Nov 03 15:36:41.732510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.732435 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z" event={"ID":"3a8227fd-da10-4bd8-b86f-5f7fc48ef40c","Type":"ContainerDied","Data":"7a1fb69e11b1a9b059c8e1f2e1d03d1287f28d2c2e34e7680112bf4b1dfa6754"} Nov 03 15:36:41.734209 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.734188 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" Nov 03 15:36:41.734305 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.734190 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx" event={"ID":"a589a74b-03fc-4a08-bd68-9ad4066f7ce1","Type":"ContainerDied","Data":"19f541ddaa81305865907ab2cb11317b80ed67becc496d79dc966688eb0b77b3"} Nov 03 15:36:41.735681 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.735662 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" event={"ID":"e362b475-7544-4095-a465-24905a71c6c0","Type":"ContainerDied","Data":"10f5465e8fa091ae3f6a09414e08e3d4b1541b11b2ee30fd844f34c669553ddb"} Nov 03 15:36:41.735782 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.735695 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb" Nov 03 15:36:41.742310 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.742266 4557 scope.go:122] "RemoveContainer" containerID="dc73687addedf98708f783dfaafb06080d641677899ec127943fc2b696cb2d88" Nov 03 15:36:41.753207 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.753190 4557 scope.go:122] "RemoveContainer" containerID="eb2aa0669f07bc636d3e348d49eb3be5adcb85fbabd84abcf306496776996860" Nov 03 15:36:41.762265 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.762249 4557 scope.go:122] "RemoveContainer" containerID="9387861b5b3928ba16c1e26e80a13ef6941619ab5f09fcf231a13bb8bbc39874" Nov 03 15:36:41.899873 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.899839 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8"] Nov 03 15:36:41.949692 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:41.949660 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-n2kq8"] Nov 03 15:36:42.022431 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:42.022355 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb"] Nov 03 15:36:42.034027 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:42.033999 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4vjmb"] Nov 03 15:36:42.160442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:42.160409 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z"] Nov 03 15:36:42.163526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:42.163504 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-4gw8z"] Nov 03 15:36:42.274567 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:42.274441 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx"] Nov 03 15:36:42.286193 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:42.286164 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" path="/var/lib/kubelet/pods/3a8227fd-da10-4bd8-b86f-5f7fc48ef40c/volumes" Nov 03 15:36:42.286699 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:42.286684 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53cb6efb-85fe-4c12-962c-436a99fedfa5" path="/var/lib/kubelet/pods/53cb6efb-85fe-4c12-962c-436a99fedfa5/volumes" Nov 03 15:36:42.287159 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:42.287146 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e362b475-7544-4095-a465-24905a71c6c0" path="/var/lib/kubelet/pods/e362b475-7544-4095-a465-24905a71c6c0/volumes" Nov 03 15:36:42.287754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:42.287740 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-lljkx"] Nov 03 15:36:44.167952 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:44.167919 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:44.286062 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:44.286030 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a589a74b-03fc-4a08-bd68-9ad4066f7ce1" path="/var/lib/kubelet/pods/a589a74b-03fc-4a08-bd68-9ad4066f7ce1/volumes" Nov 03 15:36:45.102839 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:45.102799 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:46.531379 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:36:46.531339 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:36:50.112735 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:50.112699 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:50.311530 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:50.311495 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:36:55.395782 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:36:55.395726 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:00.320722 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:00.320672 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:05.208384 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:05.208340 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:05.407705 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:05.407671 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:10.104076 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:10.104040 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:11.628123 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:11.628081 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:12.349969 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:12.349928 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:12.677744 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:12.677641 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:15.542924 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:15.542889 4557 scope.go:122] "RemoveContainer" containerID="396592b567338ded5d33413a31b6c1efd84e5eab95f831d6d3b131895e202367" Nov 03 15:37:15.665615 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:15.663930 4557 scope.go:122] "RemoveContainer" containerID="1fc87ef25d1448d7e55c1e801b6de90c52c64b699df93597531db2ac2e864b69" Nov 03 15:37:15.665615 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:15.664469 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:15.811909 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:15.807725 4557 scope.go:122] "RemoveContainer" containerID="9f623813d98f304cdcf4fb1cca855cbe2eb4d48e28c1d01044c870c7529055c4" Nov 03 15:37:15.932317 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:15.932291 4557 scope.go:122] "RemoveContainer" containerID="9d3aa5e271255dd09ca7098045f72270fc4c8cfb3fb1c0b12cf5418a9b3f1b23" Nov 03 15:37:16.097617 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:16.090669 4557 scope.go:122] "RemoveContainer" containerID="106db7ee9c7f143a88b3b1a64621d0dc006883378eb89da47e7fe3304ca7df4b" Nov 03 15:37:16.186634 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:16.176763 4557 scope.go:122] "RemoveContainer" containerID="6cd2d6c4a92e6ff1dd9b55f6f6a33c4e46bd22a1694f6b00caabf54f79a6e046" Nov 03 15:37:16.302607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:16.297703 4557 scope.go:122] "RemoveContainer" containerID="739c5ef391738f1ba58c771d66f9e5f08107c39e67b3d6f2ad413c865a5da2c1" Nov 03 15:37:16.346813 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:16.346783 4557 scope.go:122] "RemoveContainer" containerID="1d2bd20e15567a5820a7c7a60d4626d82d648ca995d947332bc2f42ffcef3ddb" Nov 03 15:37:16.534513 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:37:16.534474 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:37:16.945800 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:16.945765 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:17.265126 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:17.265028 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:18.643194 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:18.643153 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:19.563295 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:19.563259 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:19.641035 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:19.641005 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z"] Nov 03 15:37:19.641533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:19.641500 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" podUID="24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" containerName="istio-proxy" containerID="cri-o://053f28f60a320524f23c3ffecf5c6e7e09508eb2c395d40f01b7e3996211eb93" gracePeriod=30 Nov 03 15:37:19.701674 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:19.701635 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs"] Nov 03 15:37:19.702506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:19.702454 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" podUID="da2ce135-8cc3-4b9a-9503-856817681c6b" containerName="istio-proxy" containerID="cri-o://b00e087aa449570081dacd7e14fc1013b1a01a14639c7ef82d098ce916646b33" gracePeriod=30 Nov 03 15:37:19.938664 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:19.838879 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc"] Nov 03 15:37:19.939073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:19.939047 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj"] Nov 03 15:37:19.947000 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:19.939342 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" podUID="932d5d02-ff9f-40e8-bc1e-7f672c989547" containerName="istio-proxy" containerID="cri-o://8a44d028a5bcfc4325d17315125af479793bd364ad0bd96c03be29ff9a1b0934" gracePeriod=30 Nov 03 15:37:19.947000 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:19.944518 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4"] Nov 03 15:37:19.947000 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:19.944740 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" podUID="34f61e10-89c2-4c65-be7d-31a468a7f7e6" containerName="istio-proxy" containerID="cri-o://6c1cb72ff273f054004fe29975e0fb3f0f7243c1e09155adbd850558622f3247" gracePeriod=30 Nov 03 15:37:19.952687 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:19.949945 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t"] Nov 03 15:37:19.952687 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:19.950177 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" podUID="7ee31efd-a014-4cfa-8578-bed65d98bfea" containerName="istio-proxy" containerID="cri-o://3e72e181b8c94e8d20c11df3afeb90ea2db355bf31b370ac23bb9daae0fa5687" gracePeriod=30 Nov 03 15:37:19.953248 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:19.953223 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" podUID="18e9b467-83f6-4526-b8ae-2723410e32a3" containerName="istio-proxy" containerID="cri-o://5a22d594b4b76ecf3a0f3d919ca55a19b06a6f8fe7d762ed083faf9cac2f44af" gracePeriod=30 Nov 03 15:37:19.955442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:19.955417 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf"] Nov 03 15:37:19.955639 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:19.955615 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" podUID="11e80ae6-ae0d-4857-b490-ed829f3b1412" containerName="istio-proxy" containerID="cri-o://4afc8d4cc8107a00d67d79d966dc4d75a6424485c8dd77e33cd3d23f404d8afb" gracePeriod=30 Nov 03 15:37:19.966644 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:19.966624 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn"] Nov 03 15:37:19.966998 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:19.966940 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" podUID="56dd9a2a-8c82-42fa-aa29-8fd331733f3a" containerName="istio-proxy" containerID="cri-o://0c13065666797a72e008ce9ccac5f6263c65459b0dcba7e1edea5a4ad215b547" gracePeriod=30 Nov 03 15:37:20.110625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:20.110573 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:20.327644 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:20.325996 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:20.524774 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:20.524733 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:21.448355 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:21.448275 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:21.848487 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:21.848446 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:37:21.848708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:21.848557 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" podUID="34f61e10-89c2-4c65-be7d-31a468a7f7e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:37:22.247150 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:22.247061 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:37:22.247304 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:22.247133 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" podUID="18e9b467-83f6-4526-b8ae-2723410e32a3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:37:24.058384 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:24.058305 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:24.257703 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:24.257668 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:24.479087 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:24.479018 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:24.754039 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:24.754001 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:37:24.754256 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:24.754072 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" podUID="932d5d02-ff9f-40e8-bc1e-7f672c989547" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:37:24.979157 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:24.979125 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:37:25.089833 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.089806 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da2ce135-8cc3-4b9a-9503-856817681c6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2ce135-8cc3-4b9a-9503-856817681c6b-istiod-ca-cert\") pod \"da2ce135-8cc3-4b9a-9503-856817681c6b\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " Nov 03 15:37:25.090252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.089920 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-token\" (UniqueName: \"kubernetes.io/projected/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-token\") pod \"da2ce135-8cc3-4b9a-9503-856817681c6b\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " Nov 03 15:37:25.090252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.089970 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-credential-socket\") pod \"da2ce135-8cc3-4b9a-9503-856817681c6b\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " Nov 03 15:37:25.090252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.090052 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da2ce135-8cc3-4b9a-9503-856817681c6b-kube-api-access-cv4rt\" (UniqueName: \"kubernetes.io/projected/da2ce135-8cc3-4b9a-9503-856817681c6b-kube-api-access-cv4rt\") pod \"da2ce135-8cc3-4b9a-9503-856817681c6b\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " Nov 03 15:37:25.090252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.090155 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-envoy\") pod \"da2ce135-8cc3-4b9a-9503-856817681c6b\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " Nov 03 15:37:25.090478 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.090253 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-workload-certs\") pod \"da2ce135-8cc3-4b9a-9503-856817681c6b\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " Nov 03 15:37:25.090478 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.090302 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-data\") pod \"da2ce135-8cc3-4b9a-9503-856817681c6b\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " Nov 03 15:37:25.090478 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.090366 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-workload-socket\") pod \"da2ce135-8cc3-4b9a-9503-856817681c6b\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " Nov 03 15:37:25.090478 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.090418 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-podinfo\") pod \"da2ce135-8cc3-4b9a-9503-856817681c6b\" (UID: \"da2ce135-8cc3-4b9a-9503-856817681c6b\") " Nov 03 15:37:25.091954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.091673 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-workload-certs" pod "da2ce135-8cc3-4b9a-9503-856817681c6b" (UID: "da2ce135-8cc3-4b9a-9503-856817681c6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.092363 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.092338 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-data" pod "da2ce135-8cc3-4b9a-9503-856817681c6b" (UID: "da2ce135-8cc3-4b9a-9503-856817681c6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.095106 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.095084 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-token" pod "da2ce135-8cc3-4b9a-9503-856817681c6b" (UID: "da2ce135-8cc3-4b9a-9503-856817681c6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:25.095388 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.095354 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2ce135-8cc3-4b9a-9503-856817681c6b-istiod-ca-cert" pod "da2ce135-8cc3-4b9a-9503-856817681c6b" (UID: "da2ce135-8cc3-4b9a-9503-856817681c6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:37:25.095597 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.095565 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-workload-socket" pod "da2ce135-8cc3-4b9a-9503-856817681c6b" (UID: "da2ce135-8cc3-4b9a-9503-856817681c6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.095765 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.095747 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-credential-socket" pod "da2ce135-8cc3-4b9a-9503-856817681c6b" (UID: "da2ce135-8cc3-4b9a-9503-856817681c6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.103821 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.103785 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-envoy" pod "da2ce135-8cc3-4b9a-9503-856817681c6b" (UID: "da2ce135-8cc3-4b9a-9503-856817681c6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.124739 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.124683 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-podinfo" pod "da2ce135-8cc3-4b9a-9503-856817681c6b" (UID: "da2ce135-8cc3-4b9a-9503-856817681c6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:37:25.128610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.126994 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2ce135-8cc3-4b9a-9503-856817681c6b-kube-api-access-cv4rt" pod "da2ce135-8cc3-4b9a-9503-856817681c6b" (UID: "da2ce135-8cc3-4b9a-9503-856817681c6b"). InnerVolumeSpecName "kube-api-access-cv4rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:25.193162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.193000 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2ce135-8cc3-4b9a-9503-856817681c6b-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.193162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.193038 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.193162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.193055 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.193162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.193069 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv4rt\" (UniqueName: \"kubernetes.io/projected/da2ce135-8cc3-4b9a-9503-856817681c6b-kube-api-access-cv4rt\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.193162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.193084 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.193162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.193097 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.193162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.193110 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.193162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.193123 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2ce135-8cc3-4b9a-9503-856817681c6b-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.193162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.193136 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2ce135-8cc3-4b9a-9503-856817681c6b-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.239753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.239345 4557 generic.go:356] "Generic (PLEG): container finished" podID="56dd9a2a-8c82-42fa-aa29-8fd331733f3a" containerID="0c13065666797a72e008ce9ccac5f6263c65459b0dcba7e1edea5a4ad215b547" exitCode=0 Nov 03 15:37:25.239753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.239427 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" event={"ID":"56dd9a2a-8c82-42fa-aa29-8fd331733f3a","Type":"ContainerDied","Data":"0c13065666797a72e008ce9ccac5f6263c65459b0dcba7e1edea5a4ad215b547"} Nov 03 15:37:25.243662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.240941 4557 generic.go:356] "Generic (PLEG): container finished" podID="da2ce135-8cc3-4b9a-9503-856817681c6b" containerID="b00e087aa449570081dacd7e14fc1013b1a01a14639c7ef82d098ce916646b33" exitCode=0 Nov 03 15:37:25.243662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.241057 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" event={"ID":"da2ce135-8cc3-4b9a-9503-856817681c6b","Type":"ContainerDied","Data":"b00e087aa449570081dacd7e14fc1013b1a01a14639c7ef82d098ce916646b33"} Nov 03 15:37:25.243662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.241073 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" Nov 03 15:37:25.243662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.241099 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs" event={"ID":"da2ce135-8cc3-4b9a-9503-856817681c6b","Type":"ContainerDied","Data":"83231bd3a931c13ba34cc794ebafe2a3987d3d7f7978932d6ff953747ab1ab1e"} Nov 03 15:37:25.243662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.241127 4557 scope.go:122] "RemoveContainer" containerID="b00e087aa449570081dacd7e14fc1013b1a01a14639c7ef82d098ce916646b33" Nov 03 15:37:25.248418 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.246399 4557 generic.go:356] "Generic (PLEG): container finished" podID="34f61e10-89c2-4c65-be7d-31a468a7f7e6" containerID="6c1cb72ff273f054004fe29975e0fb3f0f7243c1e09155adbd850558622f3247" exitCode=0 Nov 03 15:37:25.248418 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.246478 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" event={"ID":"34f61e10-89c2-4c65-be7d-31a468a7f7e6","Type":"ContainerDied","Data":"6c1cb72ff273f054004fe29975e0fb3f0f7243c1e09155adbd850558622f3247"} Nov 03 15:37:25.256113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.250349 4557 generic.go:356] "Generic (PLEG): container finished" podID="18e9b467-83f6-4526-b8ae-2723410e32a3" containerID="5a22d594b4b76ecf3a0f3d919ca55a19b06a6f8fe7d762ed083faf9cac2f44af" exitCode=0 Nov 03 15:37:25.256113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.250453 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" event={"ID":"18e9b467-83f6-4526-b8ae-2723410e32a3","Type":"ContainerDied","Data":"5a22d594b4b76ecf3a0f3d919ca55a19b06a6f8fe7d762ed083faf9cac2f44af"} Nov 03 15:37:25.256113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.252188 4557 generic.go:356] "Generic (PLEG): container finished" podID="932d5d02-ff9f-40e8-bc1e-7f672c989547" containerID="8a44d028a5bcfc4325d17315125af479793bd364ad0bd96c03be29ff9a1b0934" exitCode=0 Nov 03 15:37:25.256113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.252281 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" event={"ID":"932d5d02-ff9f-40e8-bc1e-7f672c989547","Type":"ContainerDied","Data":"8a44d028a5bcfc4325d17315125af479793bd364ad0bd96c03be29ff9a1b0934"} Nov 03 15:37:25.256113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.253849 4557 generic.go:356] "Generic (PLEG): container finished" podID="24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" containerID="053f28f60a320524f23c3ffecf5c6e7e09508eb2c395d40f01b7e3996211eb93" exitCode=0 Nov 03 15:37:25.256113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.253980 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" event={"ID":"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5","Type":"ContainerDied","Data":"053f28f60a320524f23c3ffecf5c6e7e09508eb2c395d40f01b7e3996211eb93"} Nov 03 15:37:25.256113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.254018 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" event={"ID":"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5","Type":"ContainerDied","Data":"3c23fd7dba3dd9e00fab784506c4bda53e095fbb3213950a7e872ee6cef7d44c"} Nov 03 15:37:25.256113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.254041 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c23fd7dba3dd9e00fab784506c4bda53e095fbb3213950a7e872ee6cef7d44c" Nov 03 15:37:25.256113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.255634 4557 generic.go:356] "Generic (PLEG): container finished" podID="7ee31efd-a014-4cfa-8578-bed65d98bfea" containerID="3e72e181b8c94e8d20c11df3afeb90ea2db355bf31b370ac23bb9daae0fa5687" exitCode=0 Nov 03 15:37:25.256113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.255721 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" event={"ID":"7ee31efd-a014-4cfa-8578-bed65d98bfea","Type":"ContainerDied","Data":"3e72e181b8c94e8d20c11df3afeb90ea2db355bf31b370ac23bb9daae0fa5687"} Nov 03 15:37:25.256113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.255760 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" event={"ID":"7ee31efd-a014-4cfa-8578-bed65d98bfea","Type":"ContainerDied","Data":"868de35044f665c93ec50cf3bce85b122d3cb750344add898173dcdbc077d248"} Nov 03 15:37:25.256113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.255783 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="868de35044f665c93ec50cf3bce85b122d3cb750344add898173dcdbc077d248" Nov 03 15:37:25.259199 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.258777 4557 generic.go:356] "Generic (PLEG): container finished" podID="11e80ae6-ae0d-4857-b490-ed829f3b1412" containerID="4afc8d4cc8107a00d67d79d966dc4d75a6424485c8dd77e33cd3d23f404d8afb" exitCode=0 Nov 03 15:37:25.259199 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.258860 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" event={"ID":"11e80ae6-ae0d-4857-b490-ed829f3b1412","Type":"ContainerDied","Data":"4afc8d4cc8107a00d67d79d966dc4d75a6424485c8dd77e33cd3d23f404d8afb"} Nov 03 15:37:25.261783 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.261309 4557 generic.go:356] "Generic (PLEG): container finished" podID="74678262-2f6e-41a6-aa2b-00b64aa2c545" containerID="8f98e891a9fd39a67d070baca9938c500b6259f4491099f479f7f973c4ca8b32" exitCode=0 Nov 03 15:37:25.261783 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.261380 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" event={"ID":"74678262-2f6e-41a6-aa2b-00b64aa2c545","Type":"ContainerDied","Data":"8f98e891a9fd39a67d070baca9938c500b6259f4491099f479f7f973c4ca8b32"} Nov 03 15:37:25.261951 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.261875 4557 scope.go:122] "RemoveContainer" containerID="8f98e891a9fd39a67d070baca9938c500b6259f4491099f479f7f973c4ca8b32" Nov 03 15:37:25.263838 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.263810 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:37:25.263940 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.263889 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:37:25.263940 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.263925 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:37:25.276020 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.275999 4557 scope.go:122] "RemoveContainer" containerID="b00e087aa449570081dacd7e14fc1013b1a01a14639c7ef82d098ce916646b33" Nov 03 15:37:25.276973 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:37:25.276509 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00e087aa449570081dacd7e14fc1013b1a01a14639c7ef82d098ce916646b33\": container with ID starting with b00e087aa449570081dacd7e14fc1013b1a01a14639c7ef82d098ce916646b33 not found: ID does not exist" containerID="b00e087aa449570081dacd7e14fc1013b1a01a14639c7ef82d098ce916646b33" Nov 03 15:37:25.276973 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.276547 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00e087aa449570081dacd7e14fc1013b1a01a14639c7ef82d098ce916646b33"} err="failed to get container status \"b00e087aa449570081dacd7e14fc1013b1a01a14639c7ef82d098ce916646b33\": rpc error: code = NotFound desc = could not find container \"b00e087aa449570081dacd7e14fc1013b1a01a14639c7ef82d098ce916646b33\": container with ID starting with b00e087aa449570081dacd7e14fc1013b1a01a14639c7ef82d098ce916646b33 not found: ID does not exist" Nov 03 15:37:25.287083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.287059 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:37:25.289968 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.289789 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:37:25.292052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.292032 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:37:25.374240 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.373339 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:37:25.397105 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397077 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-workload-socket\") pod \"7ee31efd-a014-4cfa-8578-bed65d98bfea\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " Nov 03 15:37:25.397574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397133 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-podinfo\") pod \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " Nov 03 15:37:25.397574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397216 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-data\") pod \"7ee31efd-a014-4cfa-8578-bed65d98bfea\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " Nov 03 15:37:25.397574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397277 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-podinfo\") pod \"18e9b467-83f6-4526-b8ae-2723410e32a3\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " Nov 03 15:37:25.397574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397353 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-workload-socket\") pod \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " Nov 03 15:37:25.397574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397372 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-workload-socket" pod "7ee31efd-a014-4cfa-8578-bed65d98bfea" (UID: "7ee31efd-a014-4cfa-8578-bed65d98bfea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.397574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397401 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee31efd-a014-4cfa-8578-bed65d98bfea-kube-api-access-nt22r\" (UniqueName: \"kubernetes.io/projected/7ee31efd-a014-4cfa-8578-bed65d98bfea-kube-api-access-nt22r\") pod \"7ee31efd-a014-4cfa-8578-bed65d98bfea\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " Nov 03 15:37:25.397574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397434 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istiod-ca-cert\") pod \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " Nov 03 15:37:25.397574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397465 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-token\" (UniqueName: \"kubernetes.io/projected/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-token\") pod \"18e9b467-83f6-4526-b8ae-2723410e32a3\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " Nov 03 15:37:25.397574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397494 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-kube-api-access-hgsll\" (UniqueName: \"kubernetes.io/projected/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-kube-api-access-hgsll\") pod \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " Nov 03 15:37:25.397574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397500 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-data" pod "7ee31efd-a014-4cfa-8578-bed65d98bfea" (UID: "7ee31efd-a014-4cfa-8578-bed65d98bfea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.397574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397535 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-credential-socket\") pod \"18e9b467-83f6-4526-b8ae-2723410e32a3\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " Nov 03 15:37:25.397574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397603 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-credential-socket\") pod \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397637 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-workload-socket" pod "24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" (UID: "24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397650 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-data\") pod \"18e9b467-83f6-4526-b8ae-2723410e32a3\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397736 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee31efd-a014-4cfa-8578-bed65d98bfea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee31efd-a014-4cfa-8578-bed65d98bfea-istiod-ca-cert\") pod \"7ee31efd-a014-4cfa-8578-bed65d98bfea\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397765 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e9b467-83f6-4526-b8ae-2723410e32a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e9b467-83f6-4526-b8ae-2723410e32a3-istiod-ca-cert\") pod \"18e9b467-83f6-4526-b8ae-2723410e32a3\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397805 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-token\" (UniqueName: \"kubernetes.io/projected/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-token\") pod \"7ee31efd-a014-4cfa-8578-bed65d98bfea\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397838 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-workload-certs\") pod \"7ee31efd-a014-4cfa-8578-bed65d98bfea\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397880 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-workload-socket\") pod \"18e9b467-83f6-4526-b8ae-2723410e32a3\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397916 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-podinfo\") pod \"7ee31efd-a014-4cfa-8578-bed65d98bfea\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397953 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-envoy\") pod \"18e9b467-83f6-4526-b8ae-2723410e32a3\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397994 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e9b467-83f6-4526-b8ae-2723410e32a3-kube-api-access-smw7x\" (UniqueName: \"kubernetes.io/projected/18e9b467-83f6-4526-b8ae-2723410e32a3-kube-api-access-smw7x\") pod \"18e9b467-83f6-4526-b8ae-2723410e32a3\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.398032 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-envoy\") pod \"7ee31efd-a014-4cfa-8578-bed65d98bfea\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.398069 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-credential-socket\") pod \"7ee31efd-a014-4cfa-8578-bed65d98bfea\" (UID: \"7ee31efd-a014-4cfa-8578-bed65d98bfea\") " Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.398162 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-token\" (UniqueName: \"kubernetes.io/projected/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-token\") pod \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.398193 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-envoy\") pod \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.398231 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-workload-certs\") pod \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.398265 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-data\") pod \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\" (UID: \"24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5\") " Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.398327 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-workload-certs\") pod \"18e9b467-83f6-4526-b8ae-2723410e32a3\" (UID: \"18e9b467-83f6-4526-b8ae-2723410e32a3\") " Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.397990 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-credential-socket" pod "18e9b467-83f6-4526-b8ae-2723410e32a3" (UID: "18e9b467-83f6-4526-b8ae-2723410e32a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.399007 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-workload-certs" pod "18e9b467-83f6-4526-b8ae-2723410e32a3" (UID: "18e9b467-83f6-4526-b8ae-2723410e32a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.399262 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-workload-certs" pod "7ee31efd-a014-4cfa-8578-bed65d98bfea" (UID: "7ee31efd-a014-4cfa-8578-bed65d98bfea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.399278 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.399297 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.399314 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.399328 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.399344 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.399539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.399398 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e9b467-83f6-4526-b8ae-2723410e32a3-istiod-ca-cert" pod "18e9b467-83f6-4526-b8ae-2723410e32a3" (UID: "18e9b467-83f6-4526-b8ae-2723410e32a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:37:25.400562 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.398302 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-credential-socket" pod "24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" (UID: "24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.400562 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.398324 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee31efd-a014-4cfa-8578-bed65d98bfea-istiod-ca-cert" pod "7ee31efd-a014-4cfa-8578-bed65d98bfea" (UID: "7ee31efd-a014-4cfa-8578-bed65d98bfea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:37:25.400562 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.398457 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-data" pod "18e9b467-83f6-4526-b8ae-2723410e32a3" (UID: "18e9b467-83f6-4526-b8ae-2723410e32a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.400562 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.398642 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istiod-ca-cert" pod "24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" (UID: "24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:37:25.400904 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.400841 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-workload-certs" pod "24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" (UID: "24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.401060 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.401030 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-data" pod "24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" (UID: "24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.401421 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.401403 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-workload-socket" pod "18e9b467-83f6-4526-b8ae-2723410e32a3" (UID: "18e9b467-83f6-4526-b8ae-2723410e32a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.401691 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.401662 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-credential-socket" pod "7ee31efd-a014-4cfa-8578-bed65d98bfea" (UID: "7ee31efd-a014-4cfa-8578-bed65d98bfea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.401818 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.401794 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-envoy" pod "7ee31efd-a014-4cfa-8578-bed65d98bfea" (UID: "7ee31efd-a014-4cfa-8578-bed65d98bfea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.402653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.402615 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-envoy" pod "18e9b467-83f6-4526-b8ae-2723410e32a3" (UID: "18e9b467-83f6-4526-b8ae-2723410e32a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.403311 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.403223 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:37:25.404256 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.404232 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-podinfo" pod "7ee31efd-a014-4cfa-8578-bed65d98bfea" (UID: "7ee31efd-a014-4cfa-8578-bed65d98bfea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:37:25.404354 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.404268 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-kube-api-access-hgsll" pod "24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" (UID: "24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5"). InnerVolumeSpecName "kube-api-access-hgsll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:25.404354 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.404301 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-token" pod "7ee31efd-a014-4cfa-8578-bed65d98bfea" (UID: "7ee31efd-a014-4cfa-8578-bed65d98bfea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:25.404440 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.404355 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-token" pod "18e9b467-83f6-4526-b8ae-2723410e32a3" (UID: "18e9b467-83f6-4526-b8ae-2723410e32a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:25.404612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.404551 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-podinfo" pod "24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" (UID: "24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:37:25.404612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.404557 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-podinfo" pod "18e9b467-83f6-4526-b8ae-2723410e32a3" (UID: "18e9b467-83f6-4526-b8ae-2723410e32a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:37:25.404787 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.404572 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee31efd-a014-4cfa-8578-bed65d98bfea-kube-api-access-nt22r" pod "7ee31efd-a014-4cfa-8578-bed65d98bfea" (UID: "7ee31efd-a014-4cfa-8578-bed65d98bfea"). InnerVolumeSpecName "kube-api-access-nt22r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:25.404860 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.404787 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e9b467-83f6-4526-b8ae-2723410e32a3-kube-api-access-smw7x" pod "18e9b467-83f6-4526-b8ae-2723410e32a3" (UID: "18e9b467-83f6-4526-b8ae-2723410e32a3"). InnerVolumeSpecName "kube-api-access-smw7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:25.405278 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.405252 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-token" pod "24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" (UID: "24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:25.405785 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.405762 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-envoy" pod "24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" (UID: "24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.411262 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.411215 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:37:25.500166 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500137 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-data\") pod \"11e80ae6-ae0d-4857-b490-ed829f3b1412\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " Nov 03 15:37:25.500293 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500189 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-podinfo\") pod \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " Nov 03 15:37:25.500293 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500276 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-credential-socket\") pod \"11e80ae6-ae0d-4857-b490-ed829f3b1412\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " Nov 03 15:37:25.500413 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500329 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-kube-api-access-cl4nd\" (UniqueName: \"kubernetes.io/projected/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-kube-api-access-cl4nd\") pod \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " Nov 03 15:37:25.500413 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500356 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istiod-ca-cert\") pod \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " Nov 03 15:37:25.500413 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500373 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-envoy\") pod \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " Nov 03 15:37:25.500413 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500404 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e80ae6-ae0d-4857-b490-ed829f3b1412-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e80ae6-ae0d-4857-b490-ed829f3b1412-istiod-ca-cert\") pod \"11e80ae6-ae0d-4857-b490-ed829f3b1412\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " Nov 03 15:37:25.500651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500441 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-data" pod "11e80ae6-ae0d-4857-b490-ed829f3b1412" (UID: "11e80ae6-ae0d-4857-b490-ed829f3b1412"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.500651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500482 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-workload-certs\") pod \"932d5d02-ff9f-40e8-bc1e-7f672c989547\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " Nov 03 15:37:25.500651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500510 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-token\" (UniqueName: \"kubernetes.io/projected/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-token\") pod \"932d5d02-ff9f-40e8-bc1e-7f672c989547\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " Nov 03 15:37:25.500651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500536 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-workload-socket\") pod \"11e80ae6-ae0d-4857-b490-ed829f3b1412\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " Nov 03 15:37:25.500651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500544 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-credential-socket" pod "11e80ae6-ae0d-4857-b490-ed829f3b1412" (UID: "11e80ae6-ae0d-4857-b490-ed829f3b1412"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.500651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500576 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-credential-socket\") pod \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " Nov 03 15:37:25.500651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500641 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-token\" (UniqueName: \"kubernetes.io/projected/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-token\") pod \"11e80ae6-ae0d-4857-b490-ed829f3b1412\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " Nov 03 15:37:25.501011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500681 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932d5d02-ff9f-40e8-bc1e-7f672c989547-kube-api-access-nvkql\" (UniqueName: \"kubernetes.io/projected/932d5d02-ff9f-40e8-bc1e-7f672c989547-kube-api-access-nvkql\") pod \"932d5d02-ff9f-40e8-bc1e-7f672c989547\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " Nov 03 15:37:25.501011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500715 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/932d5d02-ff9f-40e8-bc1e-7f672c989547-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932d5d02-ff9f-40e8-bc1e-7f672c989547-istiod-ca-cert\") pod \"932d5d02-ff9f-40e8-bc1e-7f672c989547\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " Nov 03 15:37:25.501011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500803 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-workload-certs\") pod \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " Nov 03 15:37:25.501011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500833 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-envoy\") pod \"932d5d02-ff9f-40e8-bc1e-7f672c989547\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " Nov 03 15:37:25.501011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500856 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-data\") pod \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " Nov 03 15:37:25.501011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500861 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-workload-certs" pod "932d5d02-ff9f-40e8-bc1e-7f672c989547" (UID: "932d5d02-ff9f-40e8-bc1e-7f672c989547"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.501011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500893 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-podinfo\") pod \"932d5d02-ff9f-40e8-bc1e-7f672c989547\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " Nov 03 15:37:25.501011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.500971 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e80ae6-ae0d-4857-b490-ed829f3b1412-kube-api-access-8f4nm\" (UniqueName: \"kubernetes.io/projected/11e80ae6-ae0d-4857-b490-ed829f3b1412-kube-api-access-8f4nm\") pod \"11e80ae6-ae0d-4857-b490-ed829f3b1412\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " Nov 03 15:37:25.501011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.501000 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istiod-ca-cert" pod "56dd9a2a-8c82-42fa-aa29-8fd331733f3a" (UID: "56dd9a2a-8c82-42fa-aa29-8fd331733f3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:37:25.501559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.501004 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-podinfo\") pod \"11e80ae6-ae0d-4857-b490-ed829f3b1412\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " Nov 03 15:37:25.501559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.501162 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-envoy\") pod \"11e80ae6-ae0d-4857-b490-ed829f3b1412\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " Nov 03 15:37:25.501559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.501202 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e80ae6-ae0d-4857-b490-ed829f3b1412-istiod-ca-cert" pod "11e80ae6-ae0d-4857-b490-ed829f3b1412" (UID: "11e80ae6-ae0d-4857-b490-ed829f3b1412"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:37:25.501559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.501215 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-workload-socket\") pod \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " Nov 03 15:37:25.501559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.501324 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-workload-certs\") pod \"11e80ae6-ae0d-4857-b490-ed829f3b1412\" (UID: \"11e80ae6-ae0d-4857-b490-ed829f3b1412\") " Nov 03 15:37:25.501559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.501376 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-workload-socket\") pod \"932d5d02-ff9f-40e8-bc1e-7f672c989547\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " Nov 03 15:37:25.501559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.501408 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-credential-socket\") pod \"932d5d02-ff9f-40e8-bc1e-7f672c989547\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " Nov 03 15:37:25.501559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.501478 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-token\" (UniqueName: \"kubernetes.io/projected/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-token\") pod \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\" (UID: \"56dd9a2a-8c82-42fa-aa29-8fd331733f3a\") " Nov 03 15:37:25.501559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.501533 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-data\") pod \"932d5d02-ff9f-40e8-bc1e-7f672c989547\" (UID: \"932d5d02-ff9f-40e8-bc1e-7f672c989547\") " Nov 03 15:37:25.501999 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.501568 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932d5d02-ff9f-40e8-bc1e-7f672c989547-istiod-ca-cert" pod "932d5d02-ff9f-40e8-bc1e-7f672c989547" (UID: "932d5d02-ff9f-40e8-bc1e-7f672c989547"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:37:25.502310 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502281 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-credential-socket" pod "56dd9a2a-8c82-42fa-aa29-8fd331733f3a" (UID: "56dd9a2a-8c82-42fa-aa29-8fd331733f3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.502427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502359 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502377 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee31efd-a014-4cfa-8578-bed65d98bfea-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502386 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e9b467-83f6-4526-b8ae-2723410e32a3-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502397 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502406 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502414 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502424 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502435 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502454 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502474 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smw7x\" (UniqueName: \"kubernetes.io/projected/18e9b467-83f6-4526-b8ae-2723410e32a3-kube-api-access-smw7x\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502489 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502504 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee31efd-a014-4cfa-8578-bed65d98bfea-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502517 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502528 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e80ae6-ae0d-4857-b490-ed829f3b1412-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502541 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502557 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502572 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502626 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502641 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502655 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932d5d02-ff9f-40e8-bc1e-7f672c989547-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502668 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502682 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502696 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt22r\" (UniqueName: \"kubernetes.io/projected/7ee31efd-a014-4cfa-8578-bed65d98bfea-kube-api-access-nt22r\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502712 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502729 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502745 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgsll\" (UniqueName: \"kubernetes.io/projected/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-kube-api-access-hgsll\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502758 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502772 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e9b467-83f6-4526-b8ae-2723410e32a3-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.502862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502708 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-workload-certs" pod "11e80ae6-ae0d-4857-b490-ed829f3b1412" (UID: "11e80ae6-ae0d-4857-b490-ed829f3b1412"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.504267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502891 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-workload-socket" pod "11e80ae6-ae0d-4857-b490-ed829f3b1412" (UID: "11e80ae6-ae0d-4857-b490-ed829f3b1412"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.504267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.502996 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-workload-socket" pod "932d5d02-ff9f-40e8-bc1e-7f672c989547" (UID: "932d5d02-ff9f-40e8-bc1e-7f672c989547"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.504267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.503106 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-envoy" pod "56dd9a2a-8c82-42fa-aa29-8fd331733f3a" (UID: "56dd9a2a-8c82-42fa-aa29-8fd331733f3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.504267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.503194 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-workload-socket" pod "56dd9a2a-8c82-42fa-aa29-8fd331733f3a" (UID: "56dd9a2a-8c82-42fa-aa29-8fd331733f3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.504267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.503213 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-credential-socket" pod "932d5d02-ff9f-40e8-bc1e-7f672c989547" (UID: "932d5d02-ff9f-40e8-bc1e-7f672c989547"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.504267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.503443 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-data" pod "932d5d02-ff9f-40e8-bc1e-7f672c989547" (UID: "932d5d02-ff9f-40e8-bc1e-7f672c989547"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.504267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.503484 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-data" pod "56dd9a2a-8c82-42fa-aa29-8fd331733f3a" (UID: "56dd9a2a-8c82-42fa-aa29-8fd331733f3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.504267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.503547 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-workload-certs" pod "56dd9a2a-8c82-42fa-aa29-8fd331733f3a" (UID: "56dd9a2a-8c82-42fa-aa29-8fd331733f3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.507024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.506992 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-token" pod "932d5d02-ff9f-40e8-bc1e-7f672c989547" (UID: "932d5d02-ff9f-40e8-bc1e-7f672c989547"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:25.507024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.506999 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-envoy" pod "11e80ae6-ae0d-4857-b490-ed829f3b1412" (UID: "11e80ae6-ae0d-4857-b490-ed829f3b1412"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.507147 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.507106 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e80ae6-ae0d-4857-b490-ed829f3b1412-kube-api-access-8f4nm" pod "11e80ae6-ae0d-4857-b490-ed829f3b1412" (UID: "11e80ae6-ae0d-4857-b490-ed829f3b1412"). InnerVolumeSpecName "kube-api-access-8f4nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:25.507269 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.507246 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932d5d02-ff9f-40e8-bc1e-7f672c989547-kube-api-access-nvkql" pod "932d5d02-ff9f-40e8-bc1e-7f672c989547" (UID: "932d5d02-ff9f-40e8-bc1e-7f672c989547"). InnerVolumeSpecName "kube-api-access-nvkql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:25.507471 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.507449 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-token" pod "11e80ae6-ae0d-4857-b490-ed829f3b1412" (UID: "11e80ae6-ae0d-4857-b490-ed829f3b1412"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:25.507640 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.507617 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-envoy" pod "932d5d02-ff9f-40e8-bc1e-7f672c989547" (UID: "932d5d02-ff9f-40e8-bc1e-7f672c989547"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.508013 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.507991 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-kube-api-access-cl4nd" pod "56dd9a2a-8c82-42fa-aa29-8fd331733f3a" (UID: "56dd9a2a-8c82-42fa-aa29-8fd331733f3a"). InnerVolumeSpecName "kube-api-access-cl4nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:25.508052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.508013 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-podinfo" pod "11e80ae6-ae0d-4857-b490-ed829f3b1412" (UID: "11e80ae6-ae0d-4857-b490-ed829f3b1412"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:37:25.508758 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.508734 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-token" pod "56dd9a2a-8c82-42fa-aa29-8fd331733f3a" (UID: "56dd9a2a-8c82-42fa-aa29-8fd331733f3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:25.509805 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.509779 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-podinfo" pod "56dd9a2a-8c82-42fa-aa29-8fd331733f3a" (UID: "56dd9a2a-8c82-42fa-aa29-8fd331733f3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:37:25.510018 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.509997 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-podinfo" pod "932d5d02-ff9f-40e8-bc1e-7f672c989547" (UID: "932d5d02-ff9f-40e8-bc1e-7f672c989547"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:37:25.539295 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.539261 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs"] Nov 03 15:37:25.542361 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.542332 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-pndfs"] Nov 03 15:37:25.604414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604382 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f4nm\" (UniqueName: \"kubernetes.io/projected/11e80ae6-ae0d-4857-b490-ed829f3b1412-kube-api-access-8f4nm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604412 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604422 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604432 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604441 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604452 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604466 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604477 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604485 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604494 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604502 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl4nd\" (UniqueName: \"kubernetes.io/projected/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-kube-api-access-cl4nd\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604510 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604519 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604530 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e80ae6-ae0d-4857-b490-ed829f3b1412-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604544 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604555 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e80ae6-ae0d-4857-b490-ed829f3b1412-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604563 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvkql\" (UniqueName: \"kubernetes.io/projected/932d5d02-ff9f-40e8-bc1e-7f672c989547-kube-api-access-nvkql\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604571 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604594 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604608 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56dd9a2a-8c82-42fa-aa29-8fd331733f3a-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.604737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.604616 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932d5d02-ff9f-40e8-bc1e-7f672c989547-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.633233 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.633208 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:37:25.705712 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.705671 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-credential-socket\") pod \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " Nov 03 15:37:25.705921 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.705831 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-workload-certs\") pod \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " Nov 03 15:37:25.705921 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.705902 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-token\" (UniqueName: \"kubernetes.io/projected/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-token\") pod \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " Nov 03 15:37:25.706061 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.705989 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f61e10-89c2-4c65-be7d-31a468a7f7e6-kube-api-access-cv7r5\" (UniqueName: \"kubernetes.io/projected/34f61e10-89c2-4c65-be7d-31a468a7f7e6-kube-api-access-cv7r5\") pod \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " Nov 03 15:37:25.706061 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.706001 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-credential-socket" pod "34f61e10-89c2-4c65-be7d-31a468a7f7e6" (UID: "34f61e10-89c2-4c65-be7d-31a468a7f7e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.706061 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.706035 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-envoy\") pod \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " Nov 03 15:37:25.706226 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.706066 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-podinfo\") pod \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " Nov 03 15:37:25.706226 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.706137 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-workload-socket\") pod \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " Nov 03 15:37:25.706226 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.706169 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istiod-ca-cert\") pod \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " Nov 03 15:37:25.706226 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.706192 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-data\") pod \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\" (UID: \"34f61e10-89c2-4c65-be7d-31a468a7f7e6\") " Nov 03 15:37:25.706226 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.706196 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-workload-certs" pod "34f61e10-89c2-4c65-be7d-31a468a7f7e6" (UID: "34f61e10-89c2-4c65-be7d-31a468a7f7e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.706495 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.706404 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-workload-socket" pod "34f61e10-89c2-4c65-be7d-31a468a7f7e6" (UID: "34f61e10-89c2-4c65-be7d-31a468a7f7e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.706796 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.706771 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istiod-ca-cert" pod "34f61e10-89c2-4c65-be7d-31a468a7f7e6" (UID: "34f61e10-89c2-4c65-be7d-31a468a7f7e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:37:25.706920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.706776 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.706920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.706826 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.706920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.706843 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.707130 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.707107 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-data" pod "34f61e10-89c2-4c65-be7d-31a468a7f7e6" (UID: "34f61e10-89c2-4c65-be7d-31a468a7f7e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.708362 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.708337 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-token" pod "34f61e10-89c2-4c65-be7d-31a468a7f7e6" (UID: "34f61e10-89c2-4c65-be7d-31a468a7f7e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:25.708458 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.708414 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-podinfo" pod "34f61e10-89c2-4c65-be7d-31a468a7f7e6" (UID: "34f61e10-89c2-4c65-be7d-31a468a7f7e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:37:25.708638 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.708618 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-envoy" pod "34f61e10-89c2-4c65-be7d-31a468a7f7e6" (UID: "34f61e10-89c2-4c65-be7d-31a468a7f7e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:37:25.708728 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.708672 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f61e10-89c2-4c65-be7d-31a468a7f7e6-kube-api-access-cv7r5" pod "34f61e10-89c2-4c65-be7d-31a468a7f7e6" (UID: "34f61e10-89c2-4c65-be7d-31a468a7f7e6"). InnerVolumeSpecName "kube-api-access-cv7r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:25.808020 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.807992 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.808020 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.808020 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.808223 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.808031 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.808223 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.808040 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.808223 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.808049 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f61e10-89c2-4c65-be7d-31a468a7f7e6-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:25.808223 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:25.808062 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv7r5\" (UniqueName: \"kubernetes.io/projected/34f61e10-89c2-4c65-be7d-31a468a7f7e6-kube-api-access-cv7r5\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:26.209910 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.209826 4557 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:37:26.209910 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.209893 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" podUID="24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:37:26.241290 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.241254 4557 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:37:26.241425 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.241314 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" podUID="56dd9a2a-8c82-42fa-aa29-8fd331733f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:37:26.273521 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.273494 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" Nov 03 15:37:26.273799 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.273513 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf" event={"ID":"11e80ae6-ae0d-4857-b490-ed829f3b1412","Type":"ContainerDied","Data":"ece96c5ade6bee976ddef78577147c236bfaaced011f733a7315bb456379aa45"} Nov 03 15:37:26.273919 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.273885 4557 scope.go:122] "RemoveContainer" containerID="4afc8d4cc8107a00d67d79d966dc4d75a6424485c8dd77e33cd3d23f404d8afb" Nov 03 15:37:26.287430 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.276991 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" event={"ID":"74678262-2f6e-41a6-aa2b-00b64aa2c545","Type":"ContainerStarted","Data":"76b0eee5f7af4f60a5c71c07eea99dddc70a9c1a81fc2243cc34c0c889c9ee1f"} Nov 03 15:37:26.314007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.313981 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" Nov 03 15:37:26.315229 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.315192 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da2ce135-8cc3-4b9a-9503-856817681c6b" path="/var/lib/kubelet/pods/da2ce135-8cc3-4b9a-9503-856817681c6b/volumes" Nov 03 15:37:26.317132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.317103 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn" event={"ID":"56dd9a2a-8c82-42fa-aa29-8fd331733f3a","Type":"ContainerDied","Data":"06e8211618559fae200afb5f62e331e6d14735852e53a6f769da8ee37beb7a0c"} Nov 03 15:37:26.320447 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.320423 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" event={"ID":"34f61e10-89c2-4c65-be7d-31a468a7f7e6","Type":"ContainerDied","Data":"ff4419c13b322d3d111d29a9960b6d4c228adf9c24acdb473d37af2bf08c26e5"} Nov 03 15:37:26.320559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.320542 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4" Nov 03 15:37:26.330756 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.324262 4557 scope.go:122] "RemoveContainer" containerID="0c13065666797a72e008ce9ccac5f6263c65459b0dcba7e1edea5a4ad215b547" Nov 03 15:37:26.332190 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.331997 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" Nov 03 15:37:26.332190 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.332022 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj" event={"ID":"18e9b467-83f6-4526-b8ae-2723410e32a3","Type":"ContainerDied","Data":"f498b6cdcf0474592e933bf4a885980ac09c51d6bcecd1534d1061461976d592"} Nov 03 15:37:26.334423 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.333840 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" event={"ID":"932d5d02-ff9f-40e8-bc1e-7f672c989547","Type":"ContainerDied","Data":"372bd0761f7cb12e749640c501e264fbe269a057bbcf5edf9d690b6e9ca7ae90"} Nov 03 15:37:26.334423 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.333868 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc" Nov 03 15:37:26.334423 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.333885 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t" Nov 03 15:37:26.334423 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.334023 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z" Nov 03 15:37:26.352943 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.352925 4557 scope.go:122] "RemoveContainer" containerID="6c1cb72ff273f054004fe29975e0fb3f0f7243c1e09155adbd850558622f3247" Nov 03 15:37:26.364557 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.364538 4557 scope.go:122] "RemoveContainer" containerID="5a22d594b4b76ecf3a0f3d919ca55a19b06a6f8fe7d762ed083faf9cac2f44af" Nov 03 15:37:26.373517 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.373495 4557 scope.go:122] "RemoveContainer" containerID="8a44d028a5bcfc4325d17315125af479793bd364ad0bd96c03be29ff9a1b0934" Nov 03 15:37:26.532798 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.532692 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf"] Nov 03 15:37:26.533800 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.533751 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-clxvf"] Nov 03 15:37:26.659559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.659524 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:37:26.659785 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.659762 4557 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 03 15:37:26.659892 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.659816 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" podUID="74678262-2f6e-41a6-aa2b-00b64aa2c545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 03 15:37:26.676411 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.676387 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:26.807728 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.807691 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn"] Nov 03 15:37:26.810534 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.810511 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-5sxkn"] Nov 03 15:37:26.905348 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.905312 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj"] Nov 03 15:37:26.910287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:26.910260 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-sclsj"] Nov 03 15:37:27.022469 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:27.022440 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t"] Nov 03 15:37:27.028122 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:27.028093 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-bg69t"] Nov 03 15:37:27.162870 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:27.162793 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z"] Nov 03 15:37:27.166985 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:27.166959 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7h74z"] Nov 03 15:37:27.302730 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:27.302662 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4"] Nov 03 15:37:27.303766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:27.303742 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-kghf4"] Nov 03 15:37:27.401605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:27.400280 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc"] Nov 03 15:37:27.404274 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:27.404210 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-7hbrc"] Nov 03 15:37:27.606506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:27.606471 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:27.659338 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:27.659307 4557 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 03 15:37:27.659502 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:27.659356 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" podUID="74678262-2f6e-41a6-aa2b-00b64aa2c545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 03 15:37:27.816353 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:27.816322 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:28.002856 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.002768 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95"] Nov 03 15:37:28.003884 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.003857 4557 cpu_manager.go:414] "containerMap: removing container" podUID="932d5d02-ff9f-40e8-bc1e-7f672c989547" containerName="istio-proxy" Nov 03 15:37:28.004032 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004017 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932d5d02-ff9f-40e8-bc1e-7f672c989547" containerName="istio-proxy" Nov 03 15:37:28.004132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004121 4557 cpu_manager.go:414] "containerMap: removing container" podUID="11e80ae6-ae0d-4857-b490-ed829f3b1412" containerName="istio-proxy" Nov 03 15:37:28.004209 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004199 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e80ae6-ae0d-4857-b490-ed829f3b1412" containerName="istio-proxy" Nov 03 15:37:28.004292 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004282 4557 cpu_manager.go:414] "containerMap: removing container" podUID="a589a74b-03fc-4a08-bd68-9ad4066f7ce1" containerName="istio-proxy" Nov 03 15:37:28.004387 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004377 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a589a74b-03fc-4a08-bd68-9ad4066f7ce1" containerName="istio-proxy" Nov 03 15:37:28.004572 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004560 4557 cpu_manager.go:414] "containerMap: removing container" podUID="56dd9a2a-8c82-42fa-aa29-8fd331733f3a" containerName="istio-proxy" Nov 03 15:37:28.004694 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004625 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56dd9a2a-8c82-42fa-aa29-8fd331733f3a" containerName="istio-proxy" Nov 03 15:37:28.004694 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004654 4557 cpu_manager.go:414] "containerMap: removing container" podUID="18e9b467-83f6-4526-b8ae-2723410e32a3" containerName="istio-proxy" Nov 03 15:37:28.004694 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004662 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e9b467-83f6-4526-b8ae-2723410e32a3" containerName="istio-proxy" Nov 03 15:37:28.004694 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004679 4557 cpu_manager.go:414] "containerMap: removing container" podUID="7ee31efd-a014-4cfa-8578-bed65d98bfea" containerName="istio-proxy" Nov 03 15:37:28.004694 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004686 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee31efd-a014-4cfa-8578-bed65d98bfea" containerName="istio-proxy" Nov 03 15:37:28.004946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004701 4557 cpu_manager.go:414] "containerMap: removing container" podUID="e362b475-7544-4095-a465-24905a71c6c0" containerName="istio-proxy" Nov 03 15:37:28.004946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004709 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e362b475-7544-4095-a465-24905a71c6c0" containerName="istio-proxy" Nov 03 15:37:28.004946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004729 4557 cpu_manager.go:414] "containerMap: removing container" podUID="24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" containerName="istio-proxy" Nov 03 15:37:28.004946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004737 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" containerName="istio-proxy" Nov 03 15:37:28.004946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004752 4557 cpu_manager.go:414] "containerMap: removing container" podUID="da2ce135-8cc3-4b9a-9503-856817681c6b" containerName="istio-proxy" Nov 03 15:37:28.004946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004759 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da2ce135-8cc3-4b9a-9503-856817681c6b" containerName="istio-proxy" Nov 03 15:37:28.004946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004770 4557 cpu_manager.go:414] "containerMap: removing container" podUID="7596d6db-b29f-4686-ada3-d58864dcc8c1" containerName="revoke-keys" Nov 03 15:37:28.004946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004777 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7596d6db-b29f-4686-ada3-d58864dcc8c1" containerName="revoke-keys" Nov 03 15:37:28.004946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004801 4557 cpu_manager.go:414] "containerMap: removing container" podUID="3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" containerName="istio-proxy" Nov 03 15:37:28.004946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004808 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" containerName="istio-proxy" Nov 03 15:37:28.004946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004824 4557 cpu_manager.go:414] "containerMap: removing container" podUID="53cb6efb-85fe-4c12-962c-436a99fedfa5" containerName="istio-proxy" Nov 03 15:37:28.004946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004833 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53cb6efb-85fe-4c12-962c-436a99fedfa5" containerName="istio-proxy" Nov 03 15:37:28.004946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004844 4557 cpu_manager.go:414] "containerMap: removing container" podUID="34f61e10-89c2-4c65-be7d-31a468a7f7e6" containerName="istio-proxy" Nov 03 15:37:28.004946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004851 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f61e10-89c2-4c65-be7d-31a468a7f7e6" containerName="istio-proxy" Nov 03 15:37:28.005616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.004987 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="e362b475-7544-4095-a465-24905a71c6c0" containerName="istio-proxy" Nov 03 15:37:28.005616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.005004 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" containerName="istio-proxy" Nov 03 15:37:28.005616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.005015 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="932d5d02-ff9f-40e8-bc1e-7f672c989547" containerName="istio-proxy" Nov 03 15:37:28.005616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.005026 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e80ae6-ae0d-4857-b490-ed829f3b1412" containerName="istio-proxy" Nov 03 15:37:28.005616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.005036 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a8227fd-da10-4bd8-b86f-5f7fc48ef40c" containerName="istio-proxy" Nov 03 15:37:28.005616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.005045 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="a589a74b-03fc-4a08-bd68-9ad4066f7ce1" containerName="istio-proxy" Nov 03 15:37:28.005616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.005054 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="da2ce135-8cc3-4b9a-9503-856817681c6b" containerName="istio-proxy" Nov 03 15:37:28.005616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.005064 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="7596d6db-b29f-4686-ada3-d58864dcc8c1" containerName="revoke-keys" Nov 03 15:37:28.005616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.005073 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="56dd9a2a-8c82-42fa-aa29-8fd331733f3a" containerName="istio-proxy" Nov 03 15:37:28.005616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.005081 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="a589a74b-03fc-4a08-bd68-9ad4066f7ce1" containerName="istio-proxy" Nov 03 15:37:28.005616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.005090 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f61e10-89c2-4c65-be7d-31a468a7f7e6" containerName="istio-proxy" Nov 03 15:37:28.005616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.005097 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e9b467-83f6-4526-b8ae-2723410e32a3" containerName="istio-proxy" Nov 03 15:37:28.005616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.005106 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="53cb6efb-85fe-4c12-962c-436a99fedfa5" containerName="istio-proxy" Nov 03 15:37:28.005616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.005114 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee31efd-a014-4cfa-8578-bed65d98bfea" containerName="istio-proxy" Nov 03 15:37:28.010434 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.010413 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95" Nov 03 15:37:28.063216 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.063186 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95"] Nov 03 15:37:28.063513 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.063490 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5d3cde\"" Nov 03 15:37:28.064196 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.063608 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5d3cde-dockercfg-lhpkk\"" Nov 03 15:37:28.166426 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.166384 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxb5r\" (UniqueName: \"kubernetes.io/projected/795e739f-2842-4d1b-bbd7-5770099cc7bb-kube-api-access-zxb5r\") pod \"payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95\" (UID: \"795e739f-2842-4d1b-bbd7-5770099cc7bb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95" Nov 03 15:37:28.166922 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.166886 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795e739f-2842-4d1b-bbd7-5770099cc7bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95\" (UID: \"795e739f-2842-4d1b-bbd7-5770099cc7bb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95" Nov 03 15:37:28.238940 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.238904 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw"] Nov 03 15:37:28.239976 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.239951 4557 cpu_manager.go:414] "containerMap: removing container" podUID="a589a74b-03fc-4a08-bd68-9ad4066f7ce1" containerName="istio-proxy" Nov 03 15:37:28.240138 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.240118 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a589a74b-03fc-4a08-bd68-9ad4066f7ce1" containerName="istio-proxy" Nov 03 15:37:28.246900 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.246879 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw" Nov 03 15:37:28.249056 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.249035 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw"] Nov 03 15:37:28.268099 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.268049 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxb5r\" (UniqueName: \"kubernetes.io/projected/795e739f-2842-4d1b-bbd7-5770099cc7bb-kube-api-access-zxb5r\") pod \"payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95\" (UID: \"795e739f-2842-4d1b-bbd7-5770099cc7bb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95" Nov 03 15:37:28.268667 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.268645 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795e739f-2842-4d1b-bbd7-5770099cc7bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95\" (UID: \"795e739f-2842-4d1b-bbd7-5770099cc7bb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95" Nov 03 15:37:28.269735 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.269706 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795e739f-2842-4d1b-bbd7-5770099cc7bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95\" (UID: \"795e739f-2842-4d1b-bbd7-5770099cc7bb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95" Nov 03 15:37:28.322606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.318607 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11e80ae6-ae0d-4857-b490-ed829f3b1412" path="/var/lib/kubelet/pods/11e80ae6-ae0d-4857-b490-ed829f3b1412/volumes" Nov 03 15:37:28.322606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.319139 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e9b467-83f6-4526-b8ae-2723410e32a3" path="/var/lib/kubelet/pods/18e9b467-83f6-4526-b8ae-2723410e32a3/volumes" Nov 03 15:37:28.322606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.319550 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5" path="/var/lib/kubelet/pods/24f3ff0c-bfe4-4b9d-b1f7-11907ca5cfe5/volumes" Nov 03 15:37:28.322606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.320315 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34f61e10-89c2-4c65-be7d-31a468a7f7e6" path="/var/lib/kubelet/pods/34f61e10-89c2-4c65-be7d-31a468a7f7e6/volumes" Nov 03 15:37:28.322606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.320744 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56dd9a2a-8c82-42fa-aa29-8fd331733f3a" path="/var/lib/kubelet/pods/56dd9a2a-8c82-42fa-aa29-8fd331733f3a/volumes" Nov 03 15:37:28.322606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.321137 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee31efd-a014-4cfa-8578-bed65d98bfea" path="/var/lib/kubelet/pods/7ee31efd-a014-4cfa-8578-bed65d98bfea/volumes" Nov 03 15:37:28.322606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.321837 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="932d5d02-ff9f-40e8-bc1e-7f672c989547" path="/var/lib/kubelet/pods/932d5d02-ff9f-40e8-bc1e-7f672c989547/volumes" Nov 03 15:37:28.330143 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.329801 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxb5r\" (UniqueName: \"kubernetes.io/projected/795e739f-2842-4d1b-bbd7-5770099cc7bb-kube-api-access-zxb5r\") pod \"payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95\" (UID: \"795e739f-2842-4d1b-bbd7-5770099cc7bb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95" Nov 03 15:37:28.371273 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.371193 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67217abc-0f32-4031-ac20-5ef4293cc64e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw\" (UID: \"67217abc-0f32-4031-ac20-5ef4293cc64e\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw" Nov 03 15:37:28.372113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.371851 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b792s\" (UniqueName: \"kubernetes.io/projected/67217abc-0f32-4031-ac20-5ef4293cc64e-kube-api-access-b792s\") pod \"payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw\" (UID: \"67217abc-0f32-4031-ac20-5ef4293cc64e\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw" Nov 03 15:37:28.473559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.473517 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67217abc-0f32-4031-ac20-5ef4293cc64e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw\" (UID: \"67217abc-0f32-4031-ac20-5ef4293cc64e\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw" Nov 03 15:37:28.473759 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.473719 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b792s\" (UniqueName: \"kubernetes.io/projected/67217abc-0f32-4031-ac20-5ef4293cc64e-kube-api-access-b792s\") pod \"payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw\" (UID: \"67217abc-0f32-4031-ac20-5ef4293cc64e\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw" Nov 03 15:37:28.474295 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.474274 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67217abc-0f32-4031-ac20-5ef4293cc64e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw\" (UID: \"67217abc-0f32-4031-ac20-5ef4293cc64e\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw" Nov 03 15:37:28.534938 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.534859 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b792s\" (UniqueName: \"kubernetes.io/projected/67217abc-0f32-4031-ac20-5ef4293cc64e-kube-api-access-b792s\") pod \"payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw\" (UID: \"67217abc-0f32-4031-ac20-5ef4293cc64e\") " pod="openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw" Nov 03 15:37:28.567890 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.567863 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw" Nov 03 15:37:28.624624 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.624594 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95" Nov 03 15:37:28.659780 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.659311 4557 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 03 15:37:28.659780 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:28.659352 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" podUID="74678262-2f6e-41a6-aa2b-00b64aa2c545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 03 15:37:29.005505 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:29.005064 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw"] Nov 03 15:37:29.107514 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:29.107474 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95"] Nov 03 15:37:29.357416 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:29.357380 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw" event={"ID":"67217abc-0f32-4031-ac20-5ef4293cc64e","Type":"ContainerStarted","Data":"43b68841eb317bc830d335887ef9a4f8e077b11ca9483509cc17af21598765b4"} Nov 03 15:37:29.357839 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:29.357435 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw" event={"ID":"67217abc-0f32-4031-ac20-5ef4293cc64e","Type":"ContainerStarted","Data":"625d3e567676f0dff46d97479bad5ccf9556284d026dd426f136eb0c51ab19ee"} Nov 03 15:37:29.358228 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:29.358211 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw" Nov 03 15:37:29.359355 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:29.359337 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95" event={"ID":"795e739f-2842-4d1b-bbd7-5770099cc7bb","Type":"ContainerStarted","Data":"951eb282aefd9acbd7bae89b875a779078950e837f984e8162c916429e229789"} Nov 03 15:37:29.359397 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:29.359372 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95" event={"ID":"795e739f-2842-4d1b-bbd7-5770099cc7bb","Type":"ContainerStarted","Data":"817fec5ad3ee9b7377da7097460a668d55b8516c47472500ae7a16e9c2ab2ec2"} Nov 03 15:37:29.359871 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:29.359856 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95" Nov 03 15:37:29.474976 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:29.474915 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw" podStartSLOduration=1.474902285 podStartE2EDuration="1.474902285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:37:29.473939124 +0000 UTC m=+1759.317060667" watchObservedRunningTime="2026-11-03 15:37:29.474902285 +0000 UTC m=+1759.318023826" Nov 03 15:37:29.597606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:29.597277 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95" podStartSLOduration=2.597255482 podStartE2EDuration="2.597255482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:37:29.591981767 +0000 UTC m=+1759.435103312" watchObservedRunningTime="2026-11-03 15:37:29.597255482 +0000 UTC m=+1759.440377024" Nov 03 15:37:29.660024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:29.659500 4557 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 03 15:37:29.660024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:29.659561 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" podUID="74678262-2f6e-41a6-aa2b-00b64aa2c545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 03 15:37:30.659192 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:30.659158 4557 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 03 15:37:30.659651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:30.659214 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" podUID="74678262-2f6e-41a6-aa2b-00b64aa2c545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 03 15:37:31.663981 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:31.663951 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:37:31.664370 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:31.664312 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:37:31.664946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:31.664927 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:37:31.718886 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:31.718854 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:36.347066 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:36.347034 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8"] Nov 03 15:37:36.350824 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:36.350808 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:37:36.393119 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:36.393091 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af721495-75e3-4b73-bb07-c291abae9b7c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8\" (UID: \"af721495-75e3-4b73-bb07-c291abae9b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:37:36.393231 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:36.393139 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af721495-75e3-4b73-bb07-c291abae9b7c-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8\" (UID: \"af721495-75e3-4b73-bb07-c291abae9b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:37:36.393288 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:36.393250 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4njcr\" (UniqueName: \"kubernetes.io/projected/af721495-75e3-4b73-bb07-c291abae9b7c-kube-api-access-4njcr\") pod \"maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8\" (UID: \"af721495-75e3-4b73-bb07-c291abae9b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:37:36.393344 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:36.393320 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af721495-75e3-4b73-bb07-c291abae9b7c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8\" (UID: \"af721495-75e3-4b73-bb07-c291abae9b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:37:36.406879 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:36.406214 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-7b3fd3-serving-cert\"" Nov 03 15:37:36.459350 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:36.459324 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8"] Nov 03 15:37:36.494150 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:36.494119 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af721495-75e3-4b73-bb07-c291abae9b7c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8\" (UID: \"af721495-75e3-4b73-bb07-c291abae9b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:37:36.494319 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:36.494230 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af721495-75e3-4b73-bb07-c291abae9b7c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8\" (UID: \"af721495-75e3-4b73-bb07-c291abae9b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:37:36.494319 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:37:36.494280 4557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-serving-cert: secret "maas-api-e2e-shared-b-w5-7b3fd3-serving-cert" not found Nov 03 15:37:36.495209 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:37:36.494362 4557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af721495-75e3-4b73-bb07-c291abae9b7c-maas-api-tls podName:af721495-75e3-4b73-bb07-c291abae9b7c nodeName:}" failed. No retries permitted until 2026-11-03 15:37:36.994340443 +0000 UTC m=+1766.837461985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af721495-75e3-4b73-bb07-c291abae9b7c-maas-api-tls") pod "maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" (UID: "af721495-75e3-4b73-bb07-c291abae9b7c") : secret "maas-api-e2e-shared-b-w5-7b3fd3-serving-cert" not found Nov 03 15:37:36.495209 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:36.494286 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af721495-75e3-4b73-bb07-c291abae9b7c-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8\" (UID: \"af721495-75e3-4b73-bb07-c291abae9b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:37:36.495209 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:36.494761 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4njcr\" (UniqueName: \"kubernetes.io/projected/af721495-75e3-4b73-bb07-c291abae9b7c-kube-api-access-4njcr\") pod \"maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8\" (UID: \"af721495-75e3-4b73-bb07-c291abae9b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:37:36.495440 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:36.495222 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af721495-75e3-4b73-bb07-c291abae9b7c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8\" (UID: \"af721495-75e3-4b73-bb07-c291abae9b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:37:36.505073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:36.497152 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af721495-75e3-4b73-bb07-c291abae9b7c-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8\" (UID: \"af721495-75e3-4b73-bb07-c291abae9b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:37:36.558906 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:36.558880 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4njcr\" (UniqueName: \"kubernetes.io/projected/af721495-75e3-4b73-bb07-c291abae9b7c-kube-api-access-4njcr\") pod \"maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8\" (UID: \"af721495-75e3-4b73-bb07-c291abae9b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:37:36.626870 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:36.626744 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:36.817032 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:36.817003 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:37.000690 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:37.000603 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af721495-75e3-4b73-bb07-c291abae9b7c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8\" (UID: \"af721495-75e3-4b73-bb07-c291abae9b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:37:37.003096 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:37.003069 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af721495-75e3-4b73-bb07-c291abae9b7c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8\" (UID: \"af721495-75e3-4b73-bb07-c291abae9b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:37:37.011449 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:37.011422 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:37.264761 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:37.264685 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:37:37.645155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:37.645127 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8"] Nov 03 15:37:37.646064 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:37:37.646033 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf721495_75e3_4b73_bb07_c291abae9b7c.slice/crio-0cc4eb1f0205726f8ec54d67e6bdbec98860b5e4c3ed3289b3ec7bfdcef5d73b WatchSource:0}: Error finding container 0cc4eb1f0205726f8ec54d67e6bdbec98860b5e4c3ed3289b3ec7bfdcef5d73b: Status 404 returned error can't find the container with id 0cc4eb1f0205726f8ec54d67e6bdbec98860b5e4c3ed3289b3ec7bfdcef5d73b Nov 03 15:37:37.647651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:37.647632 4557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:37:38.077922 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:38.077890 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg"] Nov 03 15:37:38.082519 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:38.082502 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg" Nov 03 15:37:38.115899 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:38.115871 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q2hx\" (UniqueName: \"kubernetes.io/projected/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-kube-api-access-8q2hx\") pod \"maas-api-revoke-keys-e2e-mt-42595854-gdjcg\" (UID: \"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg" Nov 03 15:37:38.116019 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:38.115929 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-42595854-gdjcg\" (UID: \"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg" Nov 03 15:37:38.116019 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:38.115998 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-42595854-gdjcg\" (UID: \"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg" Nov 03 15:37:38.135992 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:38.135967 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:37:38.186089 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:38.186062 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:37:38.188963 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:38.188940 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg"] Nov 03 15:37:38.217243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:38.217213 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q2hx\" (UniqueName: \"kubernetes.io/projected/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-kube-api-access-8q2hx\") pod \"maas-api-revoke-keys-e2e-mt-42595854-gdjcg\" (UID: \"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg" Nov 03 15:37:38.217345 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:38.217264 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-42595854-gdjcg\" (UID: \"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg" Nov 03 15:37:38.217395 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:38.217368 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-42595854-gdjcg\" (UID: \"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg" Nov 03 15:37:38.217869 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:38.217852 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-42595854-gdjcg\" (UID: \"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg" Nov 03 15:37:38.219657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:38.219637 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-42595854-gdjcg\" (UID: \"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg" Nov 03 15:37:38.280876 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:38.280847 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q2hx\" (UniqueName: \"kubernetes.io/projected/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-kube-api-access-8q2hx\") pod \"maas-api-revoke-keys-e2e-mt-42595854-gdjcg\" (UID: \"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg" Nov 03 15:37:38.395976 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:38.395903 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg" Nov 03 15:37:38.419393 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:38.419363 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" event={"ID":"af721495-75e3-4b73-bb07-c291abae9b7c","Type":"ContainerStarted","Data":"0cc4eb1f0205726f8ec54d67e6bdbec98860b5e4c3ed3289b3ec7bfdcef5d73b"} Nov 03 15:37:38.772996 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:37:38.772962 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b1764d_1a55_4c1b_8dc3_3a84a1d0ebb1.slice/crio-f2af162998b82291f52cf5b02d88d58e4ca65e4cc812ca37da20f8ac8bb1b01d WatchSource:0}: Error finding container f2af162998b82291f52cf5b02d88d58e4ca65e4cc812ca37da20f8ac8bb1b01d: Status 404 returned error can't find the container with id f2af162998b82291f52cf5b02d88d58e4ca65e4cc812ca37da20f8ac8bb1b01d Nov 03 15:37:38.773527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:38.773160 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg"] Nov 03 15:37:39.070140 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:39.070113 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v"] Nov 03 15:37:39.078192 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:39.078157 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6ee148-nv86v"] Nov 03 15:37:39.286967 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:39.286930 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:39.427086 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:39.426984 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" event={"ID":"af721495-75e3-4b73-bb07-c291abae9b7c","Type":"ContainerStarted","Data":"b7c3d0908dea5bb8e80fd87424821f7dedb3853e28a44383367e55a772b4180d"} Nov 03 15:37:39.427251 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:39.427114 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:37:39.428472 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:39.428450 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg" event={"ID":"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1","Type":"ContainerStarted","Data":"a2e63f4733f020c381d5a6003198ea5cca0a19b28024104aba9ac4f4ea74bb2f"} Nov 03 15:37:39.428590 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:39.428484 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg" event={"ID":"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1","Type":"ContainerStarted","Data":"f2af162998b82291f52cf5b02d88d58e4ca65e4cc812ca37da20f8ac8bb1b01d"} Nov 03 15:37:39.546280 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:39.546233 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" podStartSLOduration=2.681144739 podStartE2EDuration="3.546220119s" totalImagesPullingTime="865.07538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:37:39.545269782 +0000 UTC m=+1769.388391324" watchObservedRunningTime="2026-11-03 15:37:39.546220119 +0000 UTC m=+1769.389341661" Nov 03 15:37:39.663925 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:39.663810 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg" podStartSLOduration=1.663784368 podStartE2EDuration="1.663784368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:37:39.661629649 +0000 UTC m=+1769.504751193" watchObservedRunningTime="2026-11-03 15:37:39.663784368 +0000 UTC m=+1769.506905915" Nov 03 15:37:40.287070 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:40.287042 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d2e445f-70dc-4678-b24b-e7592a643418" path="/var/lib/kubelet/pods/8d2e445f-70dc-4678-b24b-e7592a643418/volumes" Nov 03 15:37:40.436126 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:40.436092 4557 generic.go:356] "Generic (PLEG): container finished" podID="f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1" containerID="a2e63f4733f020c381d5a6003198ea5cca0a19b28024104aba9ac4f4ea74bb2f" exitCode=0 Nov 03 15:37:40.436267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:40.436174 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg" event={"ID":"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1","Type":"ContainerDied","Data":"a2e63f4733f020c381d5a6003198ea5cca0a19b28024104aba9ac4f4ea74bb2f"} Nov 03 15:37:41.374708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:41.374676 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95" Nov 03 15:37:41.375116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:41.374785 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw" Nov 03 15:37:41.610304 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:41.610250 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg" Nov 03 15:37:41.668450 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:41.668372 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-maas-api-openshift-ca\") pod \"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1\" (UID: \"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1\") " Nov 03 15:37:41.668625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:41.668499 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-kube-api-access-8q2hx\" (UniqueName: \"kubernetes.io/projected/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-kube-api-access-8q2hx\") pod \"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1\" (UID: \"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1\") " Nov 03 15:37:41.668625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:41.668562 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-maas-api-xks-ca\") pod \"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1\" (UID: \"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1\") " Nov 03 15:37:41.668750 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:41.668715 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-maas-api-openshift-ca" pod "f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1" (UID: "f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:37:41.669004 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:41.668989 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:41.670596 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:41.670555 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-maas-api-xks-ca" pod "f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1" (UID: "f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:37:41.670721 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:41.670556 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-kube-api-access-8q2hx" pod "f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1" (UID: "f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1"). InnerVolumeSpecName "kube-api-access-8q2hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:37:41.771353 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:41.771303 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q2hx\" (UniqueName: \"kubernetes.io/projected/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-kube-api-access-8q2hx\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:41.771538 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:41.771372 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:37:42.200767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:42.200736 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:42.451971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:42.451367 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg" event={"ID":"f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1","Type":"ContainerDied","Data":"f2af162998b82291f52cf5b02d88d58e4ca65e4cc812ca37da20f8ac8bb1b01d"} Nov 03 15:37:42.451971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:42.451432 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2af162998b82291f52cf5b02d88d58e4ca65e4cc812ca37da20f8ac8bb1b01d" Nov 03 15:37:42.451971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:42.451502 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg" Nov 03 15:37:42.462753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:42.462721 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:45.442250 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:45.442220 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:37:46.530296 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:37:46.530256 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:37:47.224937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:47.224895 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:49.469082 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:49.469048 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:49.665968 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:49.665934 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:52.207440 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:52.207405 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:37:57.021928 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:37:57.021888 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:02.017531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:02.017483 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:02.886603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:02.886539 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:03.375298 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:03.375258 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:04.554603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:04.554361 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:04.748515 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:04.748475 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:05.768611 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:05.768568 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:06.702483 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:06.702444 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:08.294310 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:08.294276 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:08.512914 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:08.512883 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:08.716671 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:08.716565 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:08.792715 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:08.792680 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg"] Nov 03 15:38:08.844278 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:08.844239 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-42595854-gdjcg"] Nov 03 15:38:09.157792 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:09.157758 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:09.363635 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:09.363546 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:10.286472 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:10.286434 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1" path="/var/lib/kubelet/pods/f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1/volumes" Nov 03 15:38:10.454182 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:10.454148 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:10.517219 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:10.517189 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:38:10.526387 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:10.526361 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:38:10.656712 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:10.656682 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:13.947446 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:13.947411 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:14.145117 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.145076 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:14.477649 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.477619 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8"] Nov 03 15:38:14.478220 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.478197 4557 cpu_manager.go:414] "containerMap: removing container" podUID="f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1" containerName="revoke-keys" Nov 03 15:38:14.478220 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.478219 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1" containerName="revoke-keys" Nov 03 15:38:14.478434 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.478346 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b1764d-1a55-4c1b-8dc3-3a84a1d0ebb1" containerName="revoke-keys" Nov 03 15:38:14.482975 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.482953 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8" Nov 03 15:38:14.543206 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.543168 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8"] Nov 03 15:38:14.586603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.586562 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-f356bb\"" Nov 03 15:38:14.586796 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.586764 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-f356bb-dockercfg-fqk99\"" Nov 03 15:38:14.607155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.607124 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dabfda2f-cfec-47ec-a322-d5bf5eaceba6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8\" (UID: \"dabfda2f-cfec-47ec-a322-d5bf5eaceba6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8" Nov 03 15:38:14.607499 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.607467 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xqr6\" (UniqueName: \"kubernetes.io/projected/dabfda2f-cfec-47ec-a322-d5bf5eaceba6-kube-api-access-6xqr6\") pod \"payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8\" (UID: \"dabfda2f-cfec-47ec-a322-d5bf5eaceba6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8" Nov 03 15:38:14.650873 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.650839 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l"] Nov 03 15:38:14.654752 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.654731 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l" Nov 03 15:38:14.709203 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.709162 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xqr6\" (UniqueName: \"kubernetes.io/projected/dabfda2f-cfec-47ec-a322-d5bf5eaceba6-kube-api-access-6xqr6\") pod \"payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8\" (UID: \"dabfda2f-cfec-47ec-a322-d5bf5eaceba6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8" Nov 03 15:38:14.709730 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.709703 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dabfda2f-cfec-47ec-a322-d5bf5eaceba6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8\" (UID: \"dabfda2f-cfec-47ec-a322-d5bf5eaceba6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8" Nov 03 15:38:14.711274 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.710800 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dabfda2f-cfec-47ec-a322-d5bf5eaceba6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8\" (UID: \"dabfda2f-cfec-47ec-a322-d5bf5eaceba6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8" Nov 03 15:38:14.715231 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.715206 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l"] Nov 03 15:38:14.774181 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.774098 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xqr6\" (UniqueName: \"kubernetes.io/projected/dabfda2f-cfec-47ec-a322-d5bf5eaceba6-kube-api-access-6xqr6\") pod \"payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8\" (UID: \"dabfda2f-cfec-47ec-a322-d5bf5eaceba6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8" Nov 03 15:38:14.798315 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.798288 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8" Nov 03 15:38:14.811274 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.811241 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8028cf00-9a15-49a8-8551-5391baf17422-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l\" (UID: \"8028cf00-9a15-49a8-8551-5391baf17422\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l" Nov 03 15:38:14.811435 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.811412 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snvtx\" (UniqueName: \"kubernetes.io/projected/8028cf00-9a15-49a8-8551-5391baf17422-kube-api-access-snvtx\") pod \"payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l\" (UID: \"8028cf00-9a15-49a8-8551-5391baf17422\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l" Nov 03 15:38:14.912378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.912344 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snvtx\" (UniqueName: \"kubernetes.io/projected/8028cf00-9a15-49a8-8551-5391baf17422-kube-api-access-snvtx\") pod \"payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l\" (UID: \"8028cf00-9a15-49a8-8551-5391baf17422\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l" Nov 03 15:38:14.912538 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.912488 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8028cf00-9a15-49a8-8551-5391baf17422-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l\" (UID: \"8028cf00-9a15-49a8-8551-5391baf17422\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l" Nov 03 15:38:14.913187 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.913164 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8028cf00-9a15-49a8-8551-5391baf17422-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l\" (UID: \"8028cf00-9a15-49a8-8551-5391baf17422\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l" Nov 03 15:38:14.973908 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:14.973877 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snvtx\" (UniqueName: \"kubernetes.io/projected/8028cf00-9a15-49a8-8551-5391baf17422-kube-api-access-snvtx\") pod \"payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l\" (UID: \"8028cf00-9a15-49a8-8551-5391baf17422\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l" Nov 03 15:38:15.209172 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:38:15.209140 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabfda2f_cfec_47ec_a322_d5bf5eaceba6.slice/crio-284f4809923b1f27d12866bf45d4ee6a2933a8a4350263070fe96cc7c74fb947 WatchSource:0}: Error finding container 284f4809923b1f27d12866bf45d4ee6a2933a8a4350263070fe96cc7c74fb947: Status 404 returned error can't find the container with id 284f4809923b1f27d12866bf45d4ee6a2933a8a4350263070fe96cc7c74fb947 Nov 03 15:38:15.209324 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:15.209234 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8"] Nov 03 15:38:15.268607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:15.268552 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l" Nov 03 15:38:15.648379 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:15.648350 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l"] Nov 03 15:38:15.651164 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:38:15.651137 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8028cf00_9a15_49a8_8551_5391baf17422.slice/crio-f712f51f6f04fb99249561643e1ee8bfbaea1f59a63a9426b275410d5d5c1421 WatchSource:0}: Error finding container f712f51f6f04fb99249561643e1ee8bfbaea1f59a63a9426b275410d5d5c1421: Status 404 returned error can't find the container with id f712f51f6f04fb99249561643e1ee8bfbaea1f59a63a9426b275410d5d5c1421 Nov 03 15:38:15.686934 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:15.686902 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8" event={"ID":"dabfda2f-cfec-47ec-a322-d5bf5eaceba6","Type":"ContainerStarted","Data":"c99c09c6acb1deaab80fc0e1c837db0d583ee1801d96c87add054f00be48bdbd"} Nov 03 15:38:15.687070 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:15.686969 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8" event={"ID":"dabfda2f-cfec-47ec-a322-d5bf5eaceba6","Type":"ContainerStarted","Data":"284f4809923b1f27d12866bf45d4ee6a2933a8a4350263070fe96cc7c74fb947"} Nov 03 15:38:15.687070 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:15.687003 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8" Nov 03 15:38:15.688553 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:15.688528 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l" event={"ID":"8028cf00-9a15-49a8-8551-5391baf17422","Type":"ContainerStarted","Data":"f712f51f6f04fb99249561643e1ee8bfbaea1f59a63a9426b275410d5d5c1421"} Nov 03 15:38:15.806663 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:15.806604 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8" podStartSLOduration=1.806574682 podStartE2EDuration="1.806574682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:38:15.806278177 +0000 UTC m=+1805.649399733" watchObservedRunningTime="2026-11-03 15:38:15.806574682 +0000 UTC m=+1805.649696225" Nov 03 15:38:16.047267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:16.047236 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:16.395018 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:16.394905 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:16.476336 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:16.476305 4557 scope.go:122] "RemoveContainer" containerID="053f28f60a320524f23c3ffecf5c6e7e09508eb2c395d40f01b7e3996211eb93" Nov 03 15:38:16.486774 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:16.486754 4557 scope.go:122] "RemoveContainer" containerID="3e72e181b8c94e8d20c11df3afeb90ea2db355bf31b370ac23bb9daae0fa5687" Nov 03 15:38:16.537180 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:38:16.537144 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:38:16.696870 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:16.696791 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l" event={"ID":"8028cf00-9a15-49a8-8551-5391baf17422","Type":"ContainerStarted","Data":"03c0dc329fcdb8b8d1f1255d273b71bfc15f1cd939abdf1948b4f95cd64ae756"} Nov 03 15:38:16.697021 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:16.696881 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l" Nov 03 15:38:16.815240 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:16.815190 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l" podStartSLOduration=2.815177191 podStartE2EDuration="2.815177191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:38:16.814168839 +0000 UTC m=+1806.657290403" watchObservedRunningTime="2026-11-03 15:38:16.815177191 +0000 UTC m=+1806.658298732" Nov 03 15:38:17.708170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:17.708138 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:17.904758 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:17.904727 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:18.065836 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:18.065805 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s"] Nov 03 15:38:18.066053 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:18.066026 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" podUID="efec6fa1-66c0-4013-b3d0-9d08a04d8830" containerName="istio-proxy" containerID="cri-o://a3db4c1589f463eddd7a83823a2914ddfba6a7c531a87d6f83852d1d20d00e13" gracePeriod=30 Nov 03 15:38:18.106062 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:18.106022 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:18.761671 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:18.761631 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:19.383521 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:19.383489 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:19.574007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:19.573970 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:19.758143 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:19.758066 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr"] Nov 03 15:38:19.763308 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:19.763283 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:19.763788 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:19.763767 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr" Nov 03 15:38:19.820226 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:19.820195 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr"] Nov 03 15:38:19.877647 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:19.870932 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-771ea0-dockercfg-znts2\"" Nov 03 15:38:19.877647 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:19.871028 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-771ea0\"" Nov 03 15:38:19.877647 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:19.873683 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr\" (UID: \"9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr" Nov 03 15:38:19.877647 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:19.874000 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpm5d\" (UniqueName: \"kubernetes.io/projected/9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc-kube-api-access-qpm5d\") pod \"payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr\" (UID: \"9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr" Nov 03 15:38:19.930627 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:19.930546 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc"] Nov 03 15:38:19.935925 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:19.935893 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc" Nov 03 15:38:19.958459 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:19.957537 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:19.975392 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:19.975356 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr\" (UID: \"9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr" Nov 03 15:38:19.975562 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:19.975489 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpm5d\" (UniqueName: \"kubernetes.io/projected/9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc-kube-api-access-qpm5d\") pod \"payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr\" (UID: \"9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr" Nov 03 15:38:19.976142 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:19.976120 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr\" (UID: \"9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr" Nov 03 15:38:19.976257 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:19.976198 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc"] Nov 03 15:38:20.043462 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.043378 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpm5d\" (UniqueName: \"kubernetes.io/projected/9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc-kube-api-access-qpm5d\") pod \"payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr\" (UID: \"9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr" Nov 03 15:38:20.076653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.076611 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc\" (UID: \"8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc" Nov 03 15:38:20.076805 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.076746 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqwlg\" (UniqueName: \"kubernetes.io/projected/8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6-kube-api-access-gqwlg\") pod \"payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc\" (UID: \"8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc" Nov 03 15:38:20.078148 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.078128 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr" Nov 03 15:38:20.178144 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.178102 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc\" (UID: \"8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc" Nov 03 15:38:20.178315 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.178243 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqwlg\" (UniqueName: \"kubernetes.io/projected/8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6-kube-api-access-gqwlg\") pod \"payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc\" (UID: \"8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc" Nov 03 15:38:20.178908 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.178835 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc\" (UID: \"8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc" Nov 03 15:38:20.239452 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.239420 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqwlg\" (UniqueName: \"kubernetes.io/projected/8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6-kube-api-access-gqwlg\") pod \"payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc\" (UID: \"8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc" Nov 03 15:38:20.259567 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.259539 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc" Nov 03 15:38:20.462890 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.462857 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr"] Nov 03 15:38:20.467554 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:38:20.467522 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9716ef2e_74f1_44a1_a9dd_a5252b7d9dfc.slice/crio-02ead878f1d28f5df69fba0f0b7c75aa88afe790a20eeb64383ad708b9c94d2d WatchSource:0}: Error finding container 02ead878f1d28f5df69fba0f0b7c75aa88afe790a20eeb64383ad708b9c94d2d: Status 404 returned error can't find the container with id 02ead878f1d28f5df69fba0f0b7c75aa88afe790a20eeb64383ad708b9c94d2d Nov 03 15:38:20.475645 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.475623 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8"] Nov 03 15:38:20.481174 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.481158 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8" Nov 03 15:38:20.534605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.534559 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:38:20.534731 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.534642 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:38:20.535445 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.535425 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8"] Nov 03 15:38:20.585659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.584718 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv8zq\" (UniqueName: \"kubernetes.io/projected/53ef9ece-80a8-44a3-9c92-1aa529ded720-kube-api-access-qv8zq\") pod \"maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8\" (UID: \"53ef9ece-80a8-44a3-9c92-1aa529ded720\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8" Nov 03 15:38:20.585659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.584813 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ef9ece-80a8-44a3-9c92-1aa529ded720-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8\" (UID: \"53ef9ece-80a8-44a3-9c92-1aa529ded720\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8" Nov 03 15:38:20.585659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.584946 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ef9ece-80a8-44a3-9c92-1aa529ded720-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8\" (UID: \"53ef9ece-80a8-44a3-9c92-1aa529ded720\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8" Nov 03 15:38:20.635647 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.635621 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc"] Nov 03 15:38:20.637188 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:38:20.637157 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8666a3f9_e3aa_41ae_b1c2_ffdeed9d80c6.slice/crio-662f9bbe03dfdb89584c50b1f2de9b4fd2a9397a5db9c43ba3127796b499406e WatchSource:0}: Error finding container 662f9bbe03dfdb89584c50b1f2de9b4fd2a9397a5db9c43ba3127796b499406e: Status 404 returned error can't find the container with id 662f9bbe03dfdb89584c50b1f2de9b4fd2a9397a5db9c43ba3127796b499406e Nov 03 15:38:20.686489 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.686459 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv8zq\" (UniqueName: \"kubernetes.io/projected/53ef9ece-80a8-44a3-9c92-1aa529ded720-kube-api-access-qv8zq\") pod \"maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8\" (UID: \"53ef9ece-80a8-44a3-9c92-1aa529ded720\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8" Nov 03 15:38:20.686689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.686510 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ef9ece-80a8-44a3-9c92-1aa529ded720-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8\" (UID: \"53ef9ece-80a8-44a3-9c92-1aa529ded720\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8" Nov 03 15:38:20.686689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.686638 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ef9ece-80a8-44a3-9c92-1aa529ded720-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8\" (UID: \"53ef9ece-80a8-44a3-9c92-1aa529ded720\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8" Nov 03 15:38:20.687290 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.687265 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ef9ece-80a8-44a3-9c92-1aa529ded720-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8\" (UID: \"53ef9ece-80a8-44a3-9c92-1aa529ded720\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8" Nov 03 15:38:20.688783 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.688763 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ef9ece-80a8-44a3-9c92-1aa529ded720-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8\" (UID: \"53ef9ece-80a8-44a3-9c92-1aa529ded720\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8" Nov 03 15:38:20.727175 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.727146 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc" event={"ID":"8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6","Type":"ContainerStarted","Data":"eebfba88b6b008db2ce3c1f2bca8cd1624461ba96ab4a01f9bcfc5a273fe4c99"} Nov 03 15:38:20.727334 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.727198 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc" event={"ID":"8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6","Type":"ContainerStarted","Data":"662f9bbe03dfdb89584c50b1f2de9b4fd2a9397a5db9c43ba3127796b499406e"} Nov 03 15:38:20.727401 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.727348 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc" Nov 03 15:38:20.728824 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.728801 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr" event={"ID":"9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc","Type":"ContainerStarted","Data":"8993a109a82f6046f96ca18b47c7bfcf8d306e4cdd0db66327897f3300046f17"} Nov 03 15:38:20.728908 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.728840 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr" event={"ID":"9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc","Type":"ContainerStarted","Data":"02ead878f1d28f5df69fba0f0b7c75aa88afe790a20eeb64383ad708b9c94d2d"} Nov 03 15:38:20.728957 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.728906 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr" Nov 03 15:38:20.748422 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.748395 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv8zq\" (UniqueName: \"kubernetes.io/projected/53ef9ece-80a8-44a3-9c92-1aa529ded720-kube-api-access-qv8zq\") pod \"maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8\" (UID: \"53ef9ece-80a8-44a3-9c92-1aa529ded720\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8" Nov 03 15:38:20.812318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.812288 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8" Nov 03 15:38:20.849412 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.849304 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc" podStartSLOduration=1.849285701 podStartE2EDuration="1.849285701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:38:20.843638849 +0000 UTC m=+1810.686760392" watchObservedRunningTime="2026-11-03 15:38:20.849285701 +0000 UTC m=+1810.692407243" Nov 03 15:38:20.977113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:20.977052 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr" podStartSLOduration=1.977038034 podStartE2EDuration="1.977038034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:38:20.973944605 +0000 UTC m=+1810.817066162" watchObservedRunningTime="2026-11-03 15:38:20.977038034 +0000 UTC m=+1810.820159576" Nov 03 15:38:21.193005 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:21.192972 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8"] Nov 03 15:38:21.193737 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:38:21.193711 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ef9ece_80a8_44a3_9c92_1aa529ded720.slice/crio-aee17936acb8cead69e6a229939c1925284f30f0db9ccd1a710fa4ad99de83a1 WatchSource:0}: Error finding container aee17936acb8cead69e6a229939c1925284f30f0db9ccd1a710fa4ad99de83a1: Status 404 returned error can't find the container with id aee17936acb8cead69e6a229939c1925284f30f0db9ccd1a710fa4ad99de83a1 Nov 03 15:38:21.739245 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:21.739159 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8" event={"ID":"53ef9ece-80a8-44a3-9c92-1aa529ded720","Type":"ContainerStarted","Data":"38e20ce13dcd6b7041942ec344abcb73aece1560c73234ebfca46366bfe7c8f0"} Nov 03 15:38:21.739245 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:21.739217 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8" event={"ID":"53ef9ece-80a8-44a3-9c92-1aa529ded720","Type":"ContainerStarted","Data":"aee17936acb8cead69e6a229939c1925284f30f0db9ccd1a710fa4ad99de83a1"} Nov 03 15:38:21.875698 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:21.875640 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8" podStartSLOduration=1.875624771 podStartE2EDuration="1.875624771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:38:21.871040897 +0000 UTC m=+1811.714162439" watchObservedRunningTime="2026-11-03 15:38:21.875624771 +0000 UTC m=+1811.718746312" Nov 03 15:38:22.747250 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:22.747218 4557 generic.go:356] "Generic (PLEG): container finished" podID="53ef9ece-80a8-44a3-9c92-1aa529ded720" containerID="38e20ce13dcd6b7041942ec344abcb73aece1560c73234ebfca46366bfe7c8f0" exitCode=0 Nov 03 15:38:22.747431 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:22.747264 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8" event={"ID":"53ef9ece-80a8-44a3-9c92-1aa529ded720","Type":"ContainerDied","Data":"38e20ce13dcd6b7041942ec344abcb73aece1560c73234ebfca46366bfe7c8f0"} Nov 03 15:38:23.338280 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.338221 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:38:23.424277 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.424248 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-envoy\") pod \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " Nov 03 15:38:23.424450 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.424340 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efec6fa1-66c0-4013-b3d0-9d08a04d8830-kube-api-access-4cvdw\" (UniqueName: \"kubernetes.io/projected/efec6fa1-66c0-4013-b3d0-9d08a04d8830-kube-api-access-4cvdw\") pod \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " Nov 03 15:38:23.424450 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.424429 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-data\") pod \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " Nov 03 15:38:23.424548 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.424483 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istiod-ca-cert\") pod \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " Nov 03 15:38:23.424548 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.424522 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-workload-socket\") pod \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " Nov 03 15:38:23.424548 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.424543 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-credential-socket\") pod \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " Nov 03 15:38:23.424740 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.424610 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-workload-certs\") pod \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " Nov 03 15:38:23.424740 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.424657 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-token\" (UniqueName: \"kubernetes.io/projected/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-token\") pod \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " Nov 03 15:38:23.424858 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.424748 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-podinfo\") pod \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\" (UID: \"efec6fa1-66c0-4013-b3d0-9d08a04d8830\") " Nov 03 15:38:23.424858 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.424765 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-data" pod "efec6fa1-66c0-4013-b3d0-9d08a04d8830" (UID: "efec6fa1-66c0-4013-b3d0-9d08a04d8830"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:23.424946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.424926 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-workload-certs" pod "efec6fa1-66c0-4013-b3d0-9d08a04d8830" (UID: "efec6fa1-66c0-4013-b3d0-9d08a04d8830"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:23.424988 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.424942 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-workload-socket" pod "efec6fa1-66c0-4013-b3d0-9d08a04d8830" (UID: "efec6fa1-66c0-4013-b3d0-9d08a04d8830"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:23.424988 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.424970 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istiod-ca-cert" pod "efec6fa1-66c0-4013-b3d0-9d08a04d8830" (UID: "efec6fa1-66c0-4013-b3d0-9d08a04d8830"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:38:23.425102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.425083 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-credential-socket" pod "efec6fa1-66c0-4013-b3d0-9d08a04d8830" (UID: "efec6fa1-66c0-4013-b3d0-9d08a04d8830"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:23.425691 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.425665 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:23.425848 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.425825 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:23.425848 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.425840 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:23.425977 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.425850 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:23.425977 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.425859 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:23.426978 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.426955 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-podinfo" pod "efec6fa1-66c0-4013-b3d0-9d08a04d8830" (UID: "efec6fa1-66c0-4013-b3d0-9d08a04d8830"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:38:23.427099 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.427082 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-envoy" pod "efec6fa1-66c0-4013-b3d0-9d08a04d8830" (UID: "efec6fa1-66c0-4013-b3d0-9d08a04d8830"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:23.427155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.427127 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-token" pod "efec6fa1-66c0-4013-b3d0-9d08a04d8830" (UID: "efec6fa1-66c0-4013-b3d0-9d08a04d8830"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:23.427155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.427151 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efec6fa1-66c0-4013-b3d0-9d08a04d8830-kube-api-access-4cvdw" pod "efec6fa1-66c0-4013-b3d0-9d08a04d8830" (UID: "efec6fa1-66c0-4013-b3d0-9d08a04d8830"). InnerVolumeSpecName "kube-api-access-4cvdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:23.527700 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.527664 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:23.527700 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.527700 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:23.527887 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.527715 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cvdw\" (UniqueName: \"kubernetes.io/projected/efec6fa1-66c0-4013-b3d0-9d08a04d8830-kube-api-access-4cvdw\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:23.527887 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.527730 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efec6fa1-66c0-4013-b3d0-9d08a04d8830-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:23.676418 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.676341 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx"] Nov 03 15:38:23.701536 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.701509 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w"] Nov 03 15:38:23.755248 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.755214 4557 generic.go:356] "Generic (PLEG): container finished" podID="efec6fa1-66c0-4013-b3d0-9d08a04d8830" containerID="a3db4c1589f463eddd7a83823a2914ddfba6a7c531a87d6f83852d1d20d00e13" exitCode=0 Nov 03 15:38:23.755442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.755286 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" Nov 03 15:38:23.755442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.755307 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" event={"ID":"efec6fa1-66c0-4013-b3d0-9d08a04d8830","Type":"ContainerDied","Data":"a3db4c1589f463eddd7a83823a2914ddfba6a7c531a87d6f83852d1d20d00e13"} Nov 03 15:38:23.755442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.755366 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s" event={"ID":"efec6fa1-66c0-4013-b3d0-9d08a04d8830","Type":"ContainerDied","Data":"7f4c5be6a236133aa6158743ea39f467f4682e6f79ef0c73d6420c08279d9e34"} Nov 03 15:38:23.755442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.755394 4557 scope.go:122] "RemoveContainer" containerID="a3db4c1589f463eddd7a83823a2914ddfba6a7c531a87d6f83852d1d20d00e13" Nov 03 15:38:23.766568 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.766549 4557 scope.go:122] "RemoveContainer" containerID="a3db4c1589f463eddd7a83823a2914ddfba6a7c531a87d6f83852d1d20d00e13" Nov 03 15:38:23.766907 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:38:23.766886 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3db4c1589f463eddd7a83823a2914ddfba6a7c531a87d6f83852d1d20d00e13\": container with ID starting with a3db4c1589f463eddd7a83823a2914ddfba6a7c531a87d6f83852d1d20d00e13 not found: ID does not exist" containerID="a3db4c1589f463eddd7a83823a2914ddfba6a7c531a87d6f83852d1d20d00e13" Nov 03 15:38:23.767004 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.766917 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3db4c1589f463eddd7a83823a2914ddfba6a7c531a87d6f83852d1d20d00e13"} err="failed to get container status \"a3db4c1589f463eddd7a83823a2914ddfba6a7c531a87d6f83852d1d20d00e13\": rpc error: code = NotFound desc = could not find container \"a3db4c1589f463eddd7a83823a2914ddfba6a7c531a87d6f83852d1d20d00e13\": container with ID starting with a3db4c1589f463eddd7a83823a2914ddfba6a7c531a87d6f83852d1d20d00e13 not found: ID does not exist" Nov 03 15:38:23.878992 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.878936 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:23.897554 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.897530 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8" Nov 03 15:38:23.919525 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.919486 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s"] Nov 03 15:38:23.954647 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:23.952789 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-d456s"] Nov 03 15:38:24.040736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:24.040699 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53ef9ece-80a8-44a3-9c92-1aa529ded720-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ef9ece-80a8-44a3-9c92-1aa529ded720-maas-api-xks-ca\") pod \"53ef9ece-80a8-44a3-9c92-1aa529ded720\" (UID: \"53ef9ece-80a8-44a3-9c92-1aa529ded720\") " Nov 03 15:38:24.040909 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:24.040851 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53ef9ece-80a8-44a3-9c92-1aa529ded720-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ef9ece-80a8-44a3-9c92-1aa529ded720-maas-api-openshift-ca\") pod \"53ef9ece-80a8-44a3-9c92-1aa529ded720\" (UID: \"53ef9ece-80a8-44a3-9c92-1aa529ded720\") " Nov 03 15:38:24.040909 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:24.040892 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53ef9ece-80a8-44a3-9c92-1aa529ded720-kube-api-access-qv8zq\" (UniqueName: \"kubernetes.io/projected/53ef9ece-80a8-44a3-9c92-1aa529ded720-kube-api-access-qv8zq\") pod \"53ef9ece-80a8-44a3-9c92-1aa529ded720\" (UID: \"53ef9ece-80a8-44a3-9c92-1aa529ded720\") " Nov 03 15:38:24.041358 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:24.041325 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ef9ece-80a8-44a3-9c92-1aa529ded720-maas-api-openshift-ca" pod "53ef9ece-80a8-44a3-9c92-1aa529ded720" (UID: "53ef9ece-80a8-44a3-9c92-1aa529ded720"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:38:24.042873 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:24.042843 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53ef9ece-80a8-44a3-9c92-1aa529ded720-maas-api-xks-ca" pod "53ef9ece-80a8-44a3-9c92-1aa529ded720" (UID: "53ef9ece-80a8-44a3-9c92-1aa529ded720"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:38:24.043002 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:24.042976 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ef9ece-80a8-44a3-9c92-1aa529ded720-kube-api-access-qv8zq" pod "53ef9ece-80a8-44a3-9c92-1aa529ded720" (UID: "53ef9ece-80a8-44a3-9c92-1aa529ded720"). InnerVolumeSpecName "kube-api-access-qv8zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:24.142549 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:24.142518 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ef9ece-80a8-44a3-9c92-1aa529ded720-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:24.142549 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:24.142549 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ef9ece-80a8-44a3-9c92-1aa529ded720-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:24.142756 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:24.142559 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv8zq\" (UniqueName: \"kubernetes.io/projected/53ef9ece-80a8-44a3-9c92-1aa529ded720-kube-api-access-qv8zq\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:24.286359 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:24.286283 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efec6fa1-66c0-4013-b3d0-9d08a04d8830" path="/var/lib/kubelet/pods/efec6fa1-66c0-4013-b3d0-9d08a04d8830/volumes" Nov 03 15:38:24.764219 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:24.764192 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8" Nov 03 15:38:24.764600 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:24.764217 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8" event={"ID":"53ef9ece-80a8-44a3-9c92-1aa529ded720","Type":"ContainerDied","Data":"aee17936acb8cead69e6a229939c1925284f30f0db9ccd1a710fa4ad99de83a1"} Nov 03 15:38:24.764600 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:24.764272 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aee17936acb8cead69e6a229939c1925284f30f0db9ccd1a710fa4ad99de83a1" Nov 03 15:38:24.997072 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:24.997037 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:26.700670 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:26.700638 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8" Nov 03 15:38:27.707444 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:27.707412 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l" Nov 03 15:38:30.264483 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:30.264447 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:31.742048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:31.742018 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc" Nov 03 15:38:31.742469 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:31.742095 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr" Nov 03 15:38:32.423900 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:32.423863 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:32.613202 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:32.613169 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:35.004506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.004472 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j"] Nov 03 15:38:35.005081 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.005064 4557 cpu_manager.go:414] "containerMap: removing container" podUID="efec6fa1-66c0-4013-b3d0-9d08a04d8830" containerName="istio-proxy" Nov 03 15:38:35.005154 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.005082 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efec6fa1-66c0-4013-b3d0-9d08a04d8830" containerName="istio-proxy" Nov 03 15:38:35.005154 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.005099 4557 cpu_manager.go:414] "containerMap: removing container" podUID="53ef9ece-80a8-44a3-9c92-1aa529ded720" containerName="revoke-keys" Nov 03 15:38:35.005154 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.005104 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ef9ece-80a8-44a3-9c92-1aa529ded720" containerName="revoke-keys" Nov 03 15:38:35.005323 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.005193 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="efec6fa1-66c0-4013-b3d0-9d08a04d8830" containerName="istio-proxy" Nov 03 15:38:35.005323 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.005205 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ef9ece-80a8-44a3-9c92-1aa529ded720" containerName="revoke-keys" Nov 03 15:38:35.008347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.008327 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:38:35.063021 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.062986 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5d3cde-serving-cert\"" Nov 03 15:38:35.078763 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.078728 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zptzp\" (UniqueName: \"kubernetes.io/projected/4444e613-53d1-4286-be6e-fa3883b053ed-kube-api-access-zptzp\") pod \"maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j\" (UID: \"4444e613-53d1-4286-be6e-fa3883b053ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:38:35.078920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.078789 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4444e613-53d1-4286-be6e-fa3883b053ed-maas-api-tls\") pod \"maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j\" (UID: \"4444e613-53d1-4286-be6e-fa3883b053ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:38:35.078985 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.078959 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4444e613-53d1-4286-be6e-fa3883b053ed-metrics-certs\") pod \"maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j\" (UID: \"4444e613-53d1-4286-be6e-fa3883b053ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:38:35.079094 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.079075 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4444e613-53d1-4286-be6e-fa3883b053ed-openshift-service-ca\") pod \"maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j\" (UID: \"4444e613-53d1-4286-be6e-fa3883b053ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:38:35.116159 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.116131 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j"] Nov 03 15:38:35.180218 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.180181 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4444e613-53d1-4286-be6e-fa3883b053ed-openshift-service-ca\") pod \"maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j\" (UID: \"4444e613-53d1-4286-be6e-fa3883b053ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:38:35.180398 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.180308 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zptzp\" (UniqueName: \"kubernetes.io/projected/4444e613-53d1-4286-be6e-fa3883b053ed-kube-api-access-zptzp\") pod \"maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j\" (UID: \"4444e613-53d1-4286-be6e-fa3883b053ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:38:35.180398 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.180342 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4444e613-53d1-4286-be6e-fa3883b053ed-maas-api-tls\") pod \"maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j\" (UID: \"4444e613-53d1-4286-be6e-fa3883b053ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:38:35.180508 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.180439 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4444e613-53d1-4286-be6e-fa3883b053ed-metrics-certs\") pod \"maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j\" (UID: \"4444e613-53d1-4286-be6e-fa3883b053ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:38:35.180568 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:38:35.180521 4557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-serving-cert: secret "maas-api-e2e-api-a-5d3cde-serving-cert" not found Nov 03 15:38:35.180638 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:38:35.180608 4557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4444e613-53d1-4286-be6e-fa3883b053ed-maas-api-tls podName:4444e613-53d1-4286-be6e-fa3883b053ed nodeName:}" failed. No retries permitted until 2026-11-03 15:38:35.680576173 +0000 UTC m=+1825.523697697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4444e613-53d1-4286-be6e-fa3883b053ed-maas-api-tls") pod "maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" (UID: "4444e613-53d1-4286-be6e-fa3883b053ed") : secret "maas-api-e2e-api-a-5d3cde-serving-cert" not found Nov 03 15:38:35.180868 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.180847 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4444e613-53d1-4286-be6e-fa3883b053ed-openshift-service-ca\") pod \"maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j\" (UID: \"4444e613-53d1-4286-be6e-fa3883b053ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:38:35.182769 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.182747 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4444e613-53d1-4286-be6e-fa3883b053ed-metrics-certs\") pod \"maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j\" (UID: \"4444e613-53d1-4286-be6e-fa3883b053ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:38:35.242739 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.242709 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:35.247376 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.247345 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zptzp\" (UniqueName: \"kubernetes.io/projected/4444e613-53d1-4286-be6e-fa3883b053ed-kube-api-access-zptzp\") pod \"maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j\" (UID: \"4444e613-53d1-4286-be6e-fa3883b053ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:38:35.444487 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.444459 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:35.687459 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.687427 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4444e613-53d1-4286-be6e-fa3883b053ed-maas-api-tls\") pod \"maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j\" (UID: \"4444e613-53d1-4286-be6e-fa3883b053ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:38:35.689882 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.689854 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4444e613-53d1-4286-be6e-fa3883b053ed-maas-api-tls\") pod \"maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j\" (UID: \"4444e613-53d1-4286-be6e-fa3883b053ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:38:35.702496 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.702434 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:35.921637 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:35.921599 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:38:36.305456 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:36.305426 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j"] Nov 03 15:38:36.306655 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:38:36.306630 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4444e613_53d1_4286_be6e_fa3883b053ed.slice/crio-5d862348a562296ef94e7862dd7e1ddd1f053105e2be4ab9533516ca35c39907 WatchSource:0}: Error finding container 5d862348a562296ef94e7862dd7e1ddd1f053105e2be4ab9533516ca35c39907: Status 404 returned error can't find the container with id 5d862348a562296ef94e7862dd7e1ddd1f053105e2be4ab9533516ca35c39907 Nov 03 15:38:36.852648 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:36.852609 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" event={"ID":"4444e613-53d1-4286-be6e-fa3883b053ed","Type":"ContainerStarted","Data":"5d862348a562296ef94e7862dd7e1ddd1f053105e2be4ab9533516ca35c39907"} Nov 03 15:38:37.860811 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:37.860778 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" event={"ID":"4444e613-53d1-4286-be6e-fa3883b053ed","Type":"ContainerStarted","Data":"8e6334622d27f8c9893e10212c91bb2ee36a2a69a3ed6ec61247128a616ccbaa"} Nov 03 15:38:37.861222 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:37.860920 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:38:37.980631 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:37.980566 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" podStartSLOduration=3.154368385 podStartE2EDuration="3.980552376s" totalImagesPullingTime="826.183991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:38:37.977640959 +0000 UTC m=+1827.820762529" watchObservedRunningTime="2026-11-03 15:38:37.980552376 +0000 UTC m=+1827.823673918" Nov 03 15:38:38.697168 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:38.697115 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" containerID="cri-o://53792f4acb891e1b7664d7c520992cb12849d6edd7ead71cb8362e3bd3bc7485" gracePeriod=45 Nov 03 15:38:38.721621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:38.721571 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" containerID="cri-o://eff23587f09058e71fa970defacd189cdab48a47837afd6855772b2a7b1e0a15" gracePeriod=45 Nov 03 15:38:38.871092 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:38.871057 4557 generic.go:356] "Generic (PLEG): container finished" podID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerID="eff23587f09058e71fa970defacd189cdab48a47837afd6855772b2a7b1e0a15" exitCode=0 Nov 03 15:38:38.871566 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:38.871137 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" event={"ID":"b43d3a39-a80b-4a80-b04b-0ac762251667","Type":"ContainerDied","Data":"eff23587f09058e71fa970defacd189cdab48a47837afd6855772b2a7b1e0a15"} Nov 03 15:38:38.872911 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:38.872888 4557 generic.go:356] "Generic (PLEG): container finished" podID="62d54cd1-cc1f-488b-9702-739043f4753c" containerID="53792f4acb891e1b7664d7c520992cb12849d6edd7ead71cb8362e3bd3bc7485" exitCode=0 Nov 03 15:38:38.873043 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:38.872951 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" event={"ID":"62d54cd1-cc1f-488b-9702-739043f4753c","Type":"ContainerDied","Data":"53792f4acb891e1b7664d7c520992cb12849d6edd7ead71cb8362e3bd3bc7485"} Nov 03 15:38:39.089386 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.089045 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:38:39.113157 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.113099 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:38:39.148345 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.148305 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-dshm\") pod \"62d54cd1-cc1f-488b-9702-739043f4753c\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " Nov 03 15:38:39.148527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.148370 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-model-cache\") pod \"62d54cd1-cc1f-488b-9702-739043f4753c\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " Nov 03 15:38:39.148527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.148471 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62d54cd1-cc1f-488b-9702-739043f4753c-tls-certs\" (UniqueName: \"kubernetes.io/secret/62d54cd1-cc1f-488b-9702-739043f4753c-tls-certs\") pod \"62d54cd1-cc1f-488b-9702-739043f4753c\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " Nov 03 15:38:39.148675 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.148572 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-home\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-home\") pod \"62d54cd1-cc1f-488b-9702-739043f4753c\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " Nov 03 15:38:39.148675 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.148647 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d54cd1-cc1f-488b-9702-739043f4753c-kube-api-access-r4mwm\" (UniqueName: \"kubernetes.io/projected/62d54cd1-cc1f-488b-9702-739043f4753c-kube-api-access-r4mwm\") pod \"62d54cd1-cc1f-488b-9702-739043f4753c\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " Nov 03 15:38:39.148780 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.148683 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-tmp-dir\") pod \"62d54cd1-cc1f-488b-9702-739043f4753c\" (UID: \"62d54cd1-cc1f-488b-9702-739043f4753c\") " Nov 03 15:38:39.148780 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.148711 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-model-cache" pod "62d54cd1-cc1f-488b-9702-739043f4753c" (UID: "62d54cd1-cc1f-488b-9702-739043f4753c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:39.148879 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.148811 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-home" pod "62d54cd1-cc1f-488b-9702-739043f4753c" (UID: "62d54cd1-cc1f-488b-9702-739043f4753c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:39.149010 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.148987 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-tmp-dir" pod "62d54cd1-cc1f-488b-9702-739043f4753c" (UID: "62d54cd1-cc1f-488b-9702-739043f4753c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:39.149257 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.149237 4557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-model-cache\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:39.149343 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.149264 4557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-home\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:39.149343 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.149277 4557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-tmp-dir\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:39.150777 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.150758 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-dshm" pod "62d54cd1-cc1f-488b-9702-739043f4753c" (UID: "62d54cd1-cc1f-488b-9702-739043f4753c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:39.150845 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.150771 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d54cd1-cc1f-488b-9702-739043f4753c-tls-certs" pod "62d54cd1-cc1f-488b-9702-739043f4753c" (UID: "62d54cd1-cc1f-488b-9702-739043f4753c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:38:39.150956 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.150941 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d54cd1-cc1f-488b-9702-739043f4753c-kube-api-access-r4mwm" pod "62d54cd1-cc1f-488b-9702-739043f4753c" (UID: "62d54cd1-cc1f-488b-9702-739043f4753c"). InnerVolumeSpecName "kube-api-access-r4mwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:39.260531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.250115 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-tmp-dir\") pod \"b43d3a39-a80b-4a80-b04b-0ac762251667\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " Nov 03 15:38:39.260531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.250220 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-home\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-home\") pod \"b43d3a39-a80b-4a80-b04b-0ac762251667\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " Nov 03 15:38:39.260531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.250354 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-dshm\") pod \"b43d3a39-a80b-4a80-b04b-0ac762251667\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " Nov 03 15:38:39.260531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.250401 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b43d3a39-a80b-4a80-b04b-0ac762251667-kube-api-access-vt5ng\" (UniqueName: \"kubernetes.io/projected/b43d3a39-a80b-4a80-b04b-0ac762251667-kube-api-access-vt5ng\") pod \"b43d3a39-a80b-4a80-b04b-0ac762251667\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " Nov 03 15:38:39.260531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.250477 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b43d3a39-a80b-4a80-b04b-0ac762251667-tls-certs\" (UniqueName: \"kubernetes.io/secret/b43d3a39-a80b-4a80-b04b-0ac762251667-tls-certs\") pod \"b43d3a39-a80b-4a80-b04b-0ac762251667\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " Nov 03 15:38:39.260531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.250509 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-model-cache\") pod \"b43d3a39-a80b-4a80-b04b-0ac762251667\" (UID: \"b43d3a39-a80b-4a80-b04b-0ac762251667\") " Nov 03 15:38:39.260531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.250537 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-tmp-dir" pod "b43d3a39-a80b-4a80-b04b-0ac762251667" (UID: "b43d3a39-a80b-4a80-b04b-0ac762251667"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:39.260531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.251279 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4mwm\" (UniqueName: \"kubernetes.io/projected/62d54cd1-cc1f-488b-9702-739043f4753c-kube-api-access-r4mwm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:39.260531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.251304 4557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-tmp-dir\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:39.260531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.251320 4557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62d54cd1-cc1f-488b-9702-739043f4753c-dshm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:39.260531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.251334 4557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d54cd1-cc1f-488b-9702-739043f4753c-tls-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:39.260531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.251544 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-model-cache" pod "b43d3a39-a80b-4a80-b04b-0ac762251667" (UID: "b43d3a39-a80b-4a80-b04b-0ac762251667"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:39.260531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.253377 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-dshm" pod "b43d3a39-a80b-4a80-b04b-0ac762251667" (UID: "b43d3a39-a80b-4a80-b04b-0ac762251667"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:39.260531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.253509 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b43d3a39-a80b-4a80-b04b-0ac762251667-kube-api-access-vt5ng" pod "b43d3a39-a80b-4a80-b04b-0ac762251667" (UID: "b43d3a39-a80b-4a80-b04b-0ac762251667"). InnerVolumeSpecName "kube-api-access-vt5ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:39.260531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.253633 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-home" pod "b43d3a39-a80b-4a80-b04b-0ac762251667" (UID: "b43d3a39-a80b-4a80-b04b-0ac762251667"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:39.260531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.254018 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b43d3a39-a80b-4a80-b04b-0ac762251667-tls-certs" pod "b43d3a39-a80b-4a80-b04b-0ac762251667" (UID: "b43d3a39-a80b-4a80-b04b-0ac762251667"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:38:39.353502 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.353465 4557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-home\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:39.353717 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.353610 4557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-dshm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:39.353717 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.353631 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt5ng\" (UniqueName: \"kubernetes.io/projected/b43d3a39-a80b-4a80-b04b-0ac762251667-kube-api-access-vt5ng\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:39.353717 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.353640 4557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b43d3a39-a80b-4a80-b04b-0ac762251667-tls-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:39.353717 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.353650 4557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b43d3a39-a80b-4a80-b04b-0ac762251667-model-cache\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:39.887605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.884269 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" event={"ID":"b43d3a39-a80b-4a80-b04b-0ac762251667","Type":"ContainerDied","Data":"5992275e3041a0f2f80ea401638d264300df2f8eff4e15bdf6873e8c65d93eef"} Nov 03 15:38:39.887605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.884332 4557 scope.go:122] "RemoveContainer" containerID="eff23587f09058e71fa970defacd189cdab48a47837afd6855772b2a7b1e0a15" Nov 03 15:38:39.887605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.884495 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w" Nov 03 15:38:39.899977 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.891358 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" event={"ID":"62d54cd1-cc1f-488b-9702-739043f4753c","Type":"ContainerDied","Data":"8b7ac791d73a3ebc829be37934a2fe9504a675d26b52e901634765ecb66b7004"} Nov 03 15:38:39.899977 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.891459 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx" Nov 03 15:38:39.918814 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:39.918786 4557 scope.go:122] "RemoveContainer" containerID="53792f4acb891e1b7664d7c520992cb12849d6edd7ead71cb8362e3bd3bc7485" Nov 03 15:38:40.054908 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:40.054876 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx"] Nov 03 15:38:40.105435 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:40.105401 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/distinct-2-w2-f78fd6-kserve-58c458b9b7-fk5mx"] Nov 03 15:38:40.171649 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:40.171619 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w"] Nov 03 15:38:40.175918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:40.175891 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/simulator-w2-f78fd6-kserve-6d66666fcb-p8d2w"] Nov 03 15:38:40.285695 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:40.285661 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" path="/var/lib/kubelet/pods/62d54cd1-cc1f-488b-9702-739043f4753c/volumes" Nov 03 15:38:40.286118 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:40.286104 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" path="/var/lib/kubelet/pods/b43d3a39-a80b-4a80-b04b-0ac762251667/volumes" Nov 03 15:38:40.364859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:40.364782 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:43.763602 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:43.763553 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:43.878433 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:43.878407 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:38:43.949914 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:43.949883 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:45.255554 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:45.255517 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:46.530520 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:38:46.530482 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:38:50.204526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:50.204483 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:50.866011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:50.865977 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf"] Nov 03 15:38:50.866251 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:50.866208 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" podUID="82f7013a-125c-4b34-8f4b-b72f312179d8" containerName="istio-proxy" containerID="cri-o://292552c5af7a71dc650cb5e0c75fbc52d65fd3973a30aeefe7a3c8601beaa1bd" gracePeriod=30 Nov 03 15:38:51.125193 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:51.125103 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb"] Nov 03 15:38:51.125355 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:51.125328 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" podUID="5dc6676c-58c3-4ad9-9701-e76f70622bb8" containerName="istio-proxy" containerID="cri-o://25465e3e1fa3eb61806c2665563ef485207842578c7d0f744517f19ae7f7d875" gracePeriod=30 Nov 03 15:38:51.742148 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:51.742113 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc"] Nov 03 15:38:51.742674 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:51.742353 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" podUID="58f82529-9942-46d4-98dc-fa872122b973" containerName="istio-proxy" containerID="cri-o://967125a7ab4054954fe4017938510f3ad637912912db8ec2573dd3b19cb475ba" gracePeriod=30 Nov 03 15:38:52.252179 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:52.252145 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9"] Nov 03 15:38:52.252388 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:52.252351 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" podUID="d6d2a233-488a-42d3-8e7b-cd7868a6f167" containerName="istio-proxy" containerID="cri-o://7a59364f66c892b477fccc0ec046f6e1e7fc631c7e9ea39cd80d16a6fde532c7" gracePeriod=30 Nov 03 15:38:53.262687 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:53.262575 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:38:53.263058 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:53.262675 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" podUID="5dc6676c-58c3-4ad9-9701-e76f70622bb8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:38:54.828073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:54.828034 4557 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:38:54.828468 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:54.828094 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" podUID="82f7013a-125c-4b34-8f4b-b72f312179d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:38:55.139899 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:55.139767 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:38:56.002083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.002053 4557 generic.go:356] "Generic (PLEG): container finished" podID="82f7013a-125c-4b34-8f4b-b72f312179d8" containerID="292552c5af7a71dc650cb5e0c75fbc52d65fd3973a30aeefe7a3c8601beaa1bd" exitCode=0 Nov 03 15:38:56.002470 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.002132 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" event={"ID":"82f7013a-125c-4b34-8f4b-b72f312179d8","Type":"ContainerDied","Data":"292552c5af7a71dc650cb5e0c75fbc52d65fd3973a30aeefe7a3c8601beaa1bd"} Nov 03 15:38:56.232017 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.231989 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:38:56.273017 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.272984 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-workload-certs\") pod \"82f7013a-125c-4b34-8f4b-b72f312179d8\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " Nov 03 15:38:56.273195 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.273043 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-podinfo\") pod \"82f7013a-125c-4b34-8f4b-b72f312179d8\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " Nov 03 15:38:56.273195 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.273094 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-envoy\") pod \"82f7013a-125c-4b34-8f4b-b72f312179d8\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " Nov 03 15:38:56.273195 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.273124 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-workload-socket\") pod \"82f7013a-125c-4b34-8f4b-b72f312179d8\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " Nov 03 15:38:56.273195 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.273156 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-data\") pod \"82f7013a-125c-4b34-8f4b-b72f312179d8\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " Nov 03 15:38:56.273425 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.273201 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f7013a-125c-4b34-8f4b-b72f312179d8-kube-api-access-694j6\" (UniqueName: \"kubernetes.io/projected/82f7013a-125c-4b34-8f4b-b72f312179d8-kube-api-access-694j6\") pod \"82f7013a-125c-4b34-8f4b-b72f312179d8\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " Nov 03 15:38:56.273425 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.273267 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f7013a-125c-4b34-8f4b-b72f312179d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f7013a-125c-4b34-8f4b-b72f312179d8-istiod-ca-cert\") pod \"82f7013a-125c-4b34-8f4b-b72f312179d8\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " Nov 03 15:38:56.273425 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.273291 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-workload-certs" pod "82f7013a-125c-4b34-8f4b-b72f312179d8" (UID: "82f7013a-125c-4b34-8f4b-b72f312179d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:56.273425 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.273336 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-credential-socket\") pod \"82f7013a-125c-4b34-8f4b-b72f312179d8\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " Nov 03 15:38:56.273425 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.273388 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-token\" (UniqueName: \"kubernetes.io/projected/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-token\") pod \"82f7013a-125c-4b34-8f4b-b72f312179d8\" (UID: \"82f7013a-125c-4b34-8f4b-b72f312179d8\") " Nov 03 15:38:56.273725 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.273492 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-workload-socket" pod "82f7013a-125c-4b34-8f4b-b72f312179d8" (UID: "82f7013a-125c-4b34-8f4b-b72f312179d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:56.274323 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.273974 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:56.274323 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.273998 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:56.274323 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.274144 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f7013a-125c-4b34-8f4b-b72f312179d8-istiod-ca-cert" pod "82f7013a-125c-4b34-8f4b-b72f312179d8" (UID: "82f7013a-125c-4b34-8f4b-b72f312179d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:38:56.274551 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.274411 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-credential-socket" pod "82f7013a-125c-4b34-8f4b-b72f312179d8" (UID: "82f7013a-125c-4b34-8f4b-b72f312179d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:56.274551 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.274427 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-data" pod "82f7013a-125c-4b34-8f4b-b72f312179d8" (UID: "82f7013a-125c-4b34-8f4b-b72f312179d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:56.276322 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.276295 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-envoy" pod "82f7013a-125c-4b34-8f4b-b72f312179d8" (UID: "82f7013a-125c-4b34-8f4b-b72f312179d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:56.276448 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.276354 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-podinfo" pod "82f7013a-125c-4b34-8f4b-b72f312179d8" (UID: "82f7013a-125c-4b34-8f4b-b72f312179d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:38:56.276735 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.276711 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f7013a-125c-4b34-8f4b-b72f312179d8-kube-api-access-694j6" pod "82f7013a-125c-4b34-8f4b-b72f312179d8" (UID: "82f7013a-125c-4b34-8f4b-b72f312179d8"). InnerVolumeSpecName "kube-api-access-694j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:56.276936 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.276913 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-token" pod "82f7013a-125c-4b34-8f4b-b72f312179d8" (UID: "82f7013a-125c-4b34-8f4b-b72f312179d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:56.375144 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.375113 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-694j6\" (UniqueName: \"kubernetes.io/projected/82f7013a-125c-4b34-8f4b-b72f312179d8-kube-api-access-694j6\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:56.375144 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.375141 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f7013a-125c-4b34-8f4b-b72f312179d8-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:56.375144 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.375150 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:56.375435 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.375160 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:56.375435 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.375169 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:56.375435 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.375177 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:56.375435 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.375185 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f7013a-125c-4b34-8f4b-b72f312179d8-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:56.389972 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.389947 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:38:56.476523 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.476487 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-credential-socket\") pod \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " Nov 03 15:38:56.476751 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.476544 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-workload-socket\") pod \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " Nov 03 15:38:56.476751 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.476576 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-data\") pod \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " Nov 03 15:38:56.476751 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.476635 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc6676c-58c3-4ad9-9701-e76f70622bb8-kube-api-access-4w4fk\" (UniqueName: \"kubernetes.io/projected/5dc6676c-58c3-4ad9-9701-e76f70622bb8-kube-api-access-4w4fk\") pod \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " Nov 03 15:38:56.476751 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.476688 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-workload-certs\") pod \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " Nov 03 15:38:56.476751 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.476720 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-envoy\") pod \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " Nov 03 15:38:56.477020 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.476799 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-credential-socket" pod "5dc6676c-58c3-4ad9-9701-e76f70622bb8" (UID: "5dc6676c-58c3-4ad9-9701-e76f70622bb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:56.477020 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.476820 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-token\" (UniqueName: \"kubernetes.io/projected/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-token\") pod \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " Nov 03 15:38:56.477020 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.476871 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-workload-socket" pod "5dc6676c-58c3-4ad9-9701-e76f70622bb8" (UID: "5dc6676c-58c3-4ad9-9701-e76f70622bb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:56.477020 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.476969 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-podinfo\") pod \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " Nov 03 15:38:56.477254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.477035 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istiod-ca-cert\") pod \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\" (UID: \"5dc6676c-58c3-4ad9-9701-e76f70622bb8\") " Nov 03 15:38:56.477254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.477076 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-workload-certs" pod "5dc6676c-58c3-4ad9-9701-e76f70622bb8" (UID: "5dc6676c-58c3-4ad9-9701-e76f70622bb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:56.477254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.477190 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-data" pod "5dc6676c-58c3-4ad9-9701-e76f70622bb8" (UID: "5dc6676c-58c3-4ad9-9701-e76f70622bb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:56.477674 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.477642 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:56.477674 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.477667 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:56.477674 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.477667 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istiod-ca-cert" pod "5dc6676c-58c3-4ad9-9701-e76f70622bb8" (UID: "5dc6676c-58c3-4ad9-9701-e76f70622bb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:38:56.477915 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.477682 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:56.477915 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.477695 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:56.479109 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.479088 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-envoy" pod "5dc6676c-58c3-4ad9-9701-e76f70622bb8" (UID: "5dc6676c-58c3-4ad9-9701-e76f70622bb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:56.479109 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.479098 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-token" pod "5dc6676c-58c3-4ad9-9701-e76f70622bb8" (UID: "5dc6676c-58c3-4ad9-9701-e76f70622bb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:56.479720 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.479702 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-podinfo" pod "5dc6676c-58c3-4ad9-9701-e76f70622bb8" (UID: "5dc6676c-58c3-4ad9-9701-e76f70622bb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:38:56.479792 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.479731 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc6676c-58c3-4ad9-9701-e76f70622bb8-kube-api-access-4w4fk" pod "5dc6676c-58c3-4ad9-9701-e76f70622bb8" (UID: "5dc6676c-58c3-4ad9-9701-e76f70622bb8"). InnerVolumeSpecName "kube-api-access-4w4fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:56.578629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.578574 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w4fk\" (UniqueName: \"kubernetes.io/projected/5dc6676c-58c3-4ad9-9701-e76f70622bb8-kube-api-access-4w4fk\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:56.578629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.578628 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:56.578836 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.578638 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:56.578836 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.578648 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:56.578836 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:56.578657 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc6676c-58c3-4ad9-9701-e76f70622bb8-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.017395 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.012510 4557 generic.go:356] "Generic (PLEG): container finished" podID="5dc6676c-58c3-4ad9-9701-e76f70622bb8" containerID="25465e3e1fa3eb61806c2665563ef485207842578c7d0f744517f19ae7f7d875" exitCode=0 Nov 03 15:38:57.017395 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.012625 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" Nov 03 15:38:57.018885 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.012644 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" event={"ID":"5dc6676c-58c3-4ad9-9701-e76f70622bb8","Type":"ContainerDied","Data":"25465e3e1fa3eb61806c2665563ef485207842578c7d0f744517f19ae7f7d875"} Nov 03 15:38:57.018885 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.018703 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb" event={"ID":"5dc6676c-58c3-4ad9-9701-e76f70622bb8","Type":"ContainerDied","Data":"c5ad62ee8ad2b9eef752fa215d7a4f09a6e4661ae98d91c6617630854a967c29"} Nov 03 15:38:57.018885 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.018744 4557 scope.go:122] "RemoveContainer" containerID="25465e3e1fa3eb61806c2665563ef485207842578c7d0f744517f19ae7f7d875" Nov 03 15:38:57.023072 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.023051 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" Nov 03 15:38:57.023072 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.023055 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf" event={"ID":"82f7013a-125c-4b34-8f4b-b72f312179d8","Type":"ContainerDied","Data":"c62a2440c9f433888d2bc75736dca9ec3521d2568c0e703cab5cf1f2accefe3b"} Nov 03 15:38:57.024475 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.024453 4557 generic.go:356] "Generic (PLEG): container finished" podID="58f82529-9942-46d4-98dc-fa872122b973" containerID="967125a7ab4054954fe4017938510f3ad637912912db8ec2573dd3b19cb475ba" exitCode=0 Nov 03 15:38:57.024706 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.024530 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" event={"ID":"58f82529-9942-46d4-98dc-fa872122b973","Type":"ContainerDied","Data":"967125a7ab4054954fe4017938510f3ad637912912db8ec2573dd3b19cb475ba"} Nov 03 15:38:57.024706 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.024616 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" event={"ID":"58f82529-9942-46d4-98dc-fa872122b973","Type":"ContainerDied","Data":"5f2d8ca1177eeb230db785587c3be524829fd925947403d5ff2684e9a71ac00c"} Nov 03 15:38:57.024706 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.024643 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f2d8ca1177eeb230db785587c3be524829fd925947403d5ff2684e9a71ac00c" Nov 03 15:38:57.031872 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.031849 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:38:57.033389 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.033377 4557 scope.go:122] "RemoveContainer" containerID="25465e3e1fa3eb61806c2665563ef485207842578c7d0f744517f19ae7f7d875" Nov 03 15:38:57.033662 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:38:57.033644 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25465e3e1fa3eb61806c2665563ef485207842578c7d0f744517f19ae7f7d875\": container with ID starting with 25465e3e1fa3eb61806c2665563ef485207842578c7d0f744517f19ae7f7d875 not found: ID does not exist" containerID="25465e3e1fa3eb61806c2665563ef485207842578c7d0f744517f19ae7f7d875" Nov 03 15:38:57.033718 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.033670 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25465e3e1fa3eb61806c2665563ef485207842578c7d0f744517f19ae7f7d875"} err="failed to get container status \"25465e3e1fa3eb61806c2665563ef485207842578c7d0f744517f19ae7f7d875\": rpc error: code = NotFound desc = could not find container \"25465e3e1fa3eb61806c2665563ef485207842578c7d0f744517f19ae7f7d875\": container with ID starting with 25465e3e1fa3eb61806c2665563ef485207842578c7d0f744517f19ae7f7d875 not found: ID does not exist" Nov 03 15:38:57.033718 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.033689 4557 scope.go:122] "RemoveContainer" containerID="292552c5af7a71dc650cb5e0c75fbc52d65fd3973a30aeefe7a3c8601beaa1bd" Nov 03 15:38:57.084434 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.084411 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-workload-socket\") pod \"58f82529-9942-46d4-98dc-fa872122b973\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " Nov 03 15:38:57.084536 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.084522 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-istio-data\") pod \"58f82529-9942-46d4-98dc-fa872122b973\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " Nov 03 15:38:57.084588 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.084551 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f82529-9942-46d4-98dc-fa872122b973-kube-api-access-xxx46\" (UniqueName: \"kubernetes.io/projected/58f82529-9942-46d4-98dc-fa872122b973-kube-api-access-xxx46\") pod \"58f82529-9942-46d4-98dc-fa872122b973\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " Nov 03 15:38:57.084642 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.084623 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f82529-9942-46d4-98dc-fa872122b973-istio-token\" (UniqueName: \"kubernetes.io/projected/58f82529-9942-46d4-98dc-fa872122b973-istio-token\") pod \"58f82529-9942-46d4-98dc-fa872122b973\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " Nov 03 15:38:57.084685 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.084671 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-credential-socket\") pod \"58f82529-9942-46d4-98dc-fa872122b973\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " Nov 03 15:38:57.084730 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.084711 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-istio-envoy\") pod \"58f82529-9942-46d4-98dc-fa872122b973\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " Nov 03 15:38:57.084779 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.084746 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58f82529-9942-46d4-98dc-fa872122b973-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f82529-9942-46d4-98dc-fa872122b973-istio-podinfo\") pod \"58f82529-9942-46d4-98dc-fa872122b973\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " Nov 03 15:38:57.084779 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.084671 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-workload-socket" pod "58f82529-9942-46d4-98dc-fa872122b973" (UID: "58f82529-9942-46d4-98dc-fa872122b973"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:57.084878 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.084792 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-istio-data" pod "58f82529-9942-46d4-98dc-fa872122b973" (UID: "58f82529-9942-46d4-98dc-fa872122b973"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:57.084878 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.084824 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-workload-certs\") pod \"58f82529-9942-46d4-98dc-fa872122b973\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " Nov 03 15:38:57.084986 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.084883 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-credential-socket" pod "58f82529-9942-46d4-98dc-fa872122b973" (UID: "58f82529-9942-46d4-98dc-fa872122b973"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:57.084986 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.084895 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58f82529-9942-46d4-98dc-fa872122b973-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f82529-9942-46d4-98dc-fa872122b973-istiod-ca-cert\") pod \"58f82529-9942-46d4-98dc-fa872122b973\" (UID: \"58f82529-9942-46d4-98dc-fa872122b973\") " Nov 03 15:38:57.085114 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.085092 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-workload-certs" pod "58f82529-9942-46d4-98dc-fa872122b973" (UID: "58f82529-9942-46d4-98dc-fa872122b973"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:57.085264 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.085241 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f82529-9942-46d4-98dc-fa872122b973-istiod-ca-cert" pod "58f82529-9942-46d4-98dc-fa872122b973" (UID: "58f82529-9942-46d4-98dc-fa872122b973"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:38:57.085435 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.085417 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.085502 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.085442 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.085502 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.085459 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f82529-9942-46d4-98dc-fa872122b973-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.085502 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.085474 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.085502 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.085490 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.086847 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.086823 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-istio-envoy" pod "58f82529-9942-46d4-98dc-fa872122b973" (UID: "58f82529-9942-46d4-98dc-fa872122b973"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:57.086947 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.086846 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f82529-9942-46d4-98dc-fa872122b973-kube-api-access-xxx46" pod "58f82529-9942-46d4-98dc-fa872122b973" (UID: "58f82529-9942-46d4-98dc-fa872122b973"). InnerVolumeSpecName "kube-api-access-xxx46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:57.086947 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.086873 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f82529-9942-46d4-98dc-fa872122b973-istio-token" pod "58f82529-9942-46d4-98dc-fa872122b973" (UID: "58f82529-9942-46d4-98dc-fa872122b973"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:57.087041 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.086949 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58f82529-9942-46d4-98dc-fa872122b973-istio-podinfo" pod "58f82529-9942-46d4-98dc-fa872122b973" (UID: "58f82529-9942-46d4-98dc-fa872122b973"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:38:57.186605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.186557 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxx46\" (UniqueName: \"kubernetes.io/projected/58f82529-9942-46d4-98dc-fa872122b973-kube-api-access-xxx46\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.186605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.186606 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f82529-9942-46d4-98dc-fa872122b973-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.186785 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.186618 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f82529-9942-46d4-98dc-fa872122b973-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.186785 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.186627 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f82529-9942-46d4-98dc-fa872122b973-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.312688 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.312644 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb"] Nov 03 15:38:57.316521 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.316494 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-pbpzb"] Nov 03 15:38:57.433010 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.432978 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf"] Nov 03 15:38:57.438596 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.438558 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wpmkf"] Nov 03 15:38:57.491165 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.491141 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:38:57.590959 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.590933 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-data\") pod \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " Nov 03 15:38:57.591156 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.590999 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-workload-certs\") pod \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " Nov 03 15:38:57.591156 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.591036 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-credential-socket\") pod \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " Nov 03 15:38:57.591156 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.591055 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-podinfo\") pod \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " Nov 03 15:38:57.591156 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.591124 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-envoy\") pod \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " Nov 03 15:38:57.591156 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.591145 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istiod-ca-cert\") pod \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " Nov 03 15:38:57.591436 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.591188 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d2a233-488a-42d3-8e7b-cd7868a6f167-kube-api-access-qvqqh\" (UniqueName: \"kubernetes.io/projected/d6d2a233-488a-42d3-8e7b-cd7868a6f167-kube-api-access-qvqqh\") pod \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " Nov 03 15:38:57.591436 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.591261 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-workload-socket\") pod \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " Nov 03 15:38:57.591436 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.591280 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-token\" (UniqueName: \"kubernetes.io/projected/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-token\") pod \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\" (UID: \"d6d2a233-488a-42d3-8e7b-cd7868a6f167\") " Nov 03 15:38:57.591436 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.591184 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-data" pod "d6d2a233-488a-42d3-8e7b-cd7868a6f167" (UID: "d6d2a233-488a-42d3-8e7b-cd7868a6f167"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:57.591436 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.591322 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-workload-certs" pod "d6d2a233-488a-42d3-8e7b-cd7868a6f167" (UID: "d6d2a233-488a-42d3-8e7b-cd7868a6f167"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:57.591436 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.591353 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-credential-socket" pod "d6d2a233-488a-42d3-8e7b-cd7868a6f167" (UID: "d6d2a233-488a-42d3-8e7b-cd7868a6f167"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:57.591767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.591571 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-workload-socket" pod "d6d2a233-488a-42d3-8e7b-cd7868a6f167" (UID: "d6d2a233-488a-42d3-8e7b-cd7868a6f167"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:57.591767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.591671 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istiod-ca-cert" pod "d6d2a233-488a-42d3-8e7b-cd7868a6f167" (UID: "d6d2a233-488a-42d3-8e7b-cd7868a6f167"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:38:57.591968 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.591949 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.592051 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.591974 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.592051 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.591989 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.592051 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.592003 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.592051 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.592017 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.593476 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.593452 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-podinfo" pod "d6d2a233-488a-42d3-8e7b-cd7868a6f167" (UID: "d6d2a233-488a-42d3-8e7b-cd7868a6f167"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:38:57.593476 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.593470 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-token" pod "d6d2a233-488a-42d3-8e7b-cd7868a6f167" (UID: "d6d2a233-488a-42d3-8e7b-cd7868a6f167"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:57.593782 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.593760 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-envoy" pod "d6d2a233-488a-42d3-8e7b-cd7868a6f167" (UID: "d6d2a233-488a-42d3-8e7b-cd7868a6f167"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:38:57.593859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.593766 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d2a233-488a-42d3-8e7b-cd7868a6f167-kube-api-access-qvqqh" pod "d6d2a233-488a-42d3-8e7b-cd7868a6f167" (UID: "d6d2a233-488a-42d3-8e7b-cd7868a6f167"). InnerVolumeSpecName "kube-api-access-qvqqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:38:57.692844 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.692810 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.692844 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.692838 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.692844 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.692848 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d2a233-488a-42d3-8e7b-cd7868a6f167-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:57.693058 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:57.692858 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvqqh\" (UniqueName: \"kubernetes.io/projected/d6d2a233-488a-42d3-8e7b-cd7868a6f167-kube-api-access-qvqqh\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:38:58.032622 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:58.032559 4557 generic.go:356] "Generic (PLEG): container finished" podID="d6d2a233-488a-42d3-8e7b-cd7868a6f167" containerID="7a59364f66c892b477fccc0ec046f6e1e7fc631c7e9ea39cd80d16a6fde532c7" exitCode=0 Nov 03 15:38:58.033024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:58.032649 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" Nov 03 15:38:58.033024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:58.032649 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" event={"ID":"d6d2a233-488a-42d3-8e7b-cd7868a6f167","Type":"ContainerDied","Data":"7a59364f66c892b477fccc0ec046f6e1e7fc631c7e9ea39cd80d16a6fde532c7"} Nov 03 15:38:58.033024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:58.032796 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9" event={"ID":"d6d2a233-488a-42d3-8e7b-cd7868a6f167","Type":"ContainerDied","Data":"cbd53a3dfbda81be1073f1ef949b73d1b407ac168ad9c6802fc804a4864d2a7c"} Nov 03 15:38:58.033024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:58.032820 4557 scope.go:122] "RemoveContainer" containerID="7a59364f66c892b477fccc0ec046f6e1e7fc631c7e9ea39cd80d16a6fde532c7" Nov 03 15:38:58.034150 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:58.033893 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc" Nov 03 15:38:58.043050 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:58.043034 4557 scope.go:122] "RemoveContainer" containerID="7a59364f66c892b477fccc0ec046f6e1e7fc631c7e9ea39cd80d16a6fde532c7" Nov 03 15:38:58.043332 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:38:58.043309 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a59364f66c892b477fccc0ec046f6e1e7fc631c7e9ea39cd80d16a6fde532c7\": container with ID starting with 7a59364f66c892b477fccc0ec046f6e1e7fc631c7e9ea39cd80d16a6fde532c7 not found: ID does not exist" containerID="7a59364f66c892b477fccc0ec046f6e1e7fc631c7e9ea39cd80d16a6fde532c7" Nov 03 15:38:58.043426 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:58.043334 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a59364f66c892b477fccc0ec046f6e1e7fc631c7e9ea39cd80d16a6fde532c7"} err="failed to get container status \"7a59364f66c892b477fccc0ec046f6e1e7fc631c7e9ea39cd80d16a6fde532c7\": rpc error: code = NotFound desc = could not find container \"7a59364f66c892b477fccc0ec046f6e1e7fc631c7e9ea39cd80d16a6fde532c7\": container with ID starting with 7a59364f66c892b477fccc0ec046f6e1e7fc631c7e9ea39cd80d16a6fde532c7 not found: ID does not exist" Nov 03 15:38:58.200044 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:58.199963 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc"] Nov 03 15:38:58.249560 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:58.249514 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-8hnxc"] Nov 03 15:38:58.287953 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:58.287924 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58f82529-9942-46d4-98dc-fa872122b973" path="/var/lib/kubelet/pods/58f82529-9942-46d4-98dc-fa872122b973/volumes" Nov 03 15:38:58.288437 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:58.288421 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc6676c-58c3-4ad9-9701-e76f70622bb8" path="/var/lib/kubelet/pods/5dc6676c-58c3-4ad9-9701-e76f70622bb8/volumes" Nov 03 15:38:58.288911 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:58.288893 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f7013a-125c-4b34-8f4b-b72f312179d8" path="/var/lib/kubelet/pods/82f7013a-125c-4b34-8f4b-b72f312179d8/volumes" Nov 03 15:38:58.315766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:58.315743 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9"] Nov 03 15:38:58.320640 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:58.320619 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-9hzs9"] Nov 03 15:38:58.849561 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:38:58.849524 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:00.136092 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136061 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64"] Nov 03 15:39:00.136548 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136534 4557 cpu_manager.go:414] "containerMap: removing container" podUID="d6d2a233-488a-42d3-8e7b-cd7868a6f167" containerName="istio-proxy" Nov 03 15:39:00.136605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136549 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d2a233-488a-42d3-8e7b-cd7868a6f167" containerName="istio-proxy" Nov 03 15:39:00.136605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136561 4557 cpu_manager.go:414] "containerMap: removing container" podUID="82f7013a-125c-4b34-8f4b-b72f312179d8" containerName="istio-proxy" Nov 03 15:39:00.136605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136566 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f7013a-125c-4b34-8f4b-b72f312179d8" containerName="istio-proxy" Nov 03 15:39:00.136605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136591 4557 cpu_manager.go:414] "containerMap: removing container" podUID="58f82529-9942-46d4-98dc-fa872122b973" containerName="istio-proxy" Nov 03 15:39:00.136605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136597 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f82529-9942-46d4-98dc-fa872122b973" containerName="istio-proxy" Nov 03 15:39:00.136767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136609 4557 cpu_manager.go:414] "containerMap: removing container" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" Nov 03 15:39:00.136767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136615 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" Nov 03 15:39:00.136767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136640 4557 cpu_manager.go:414] "containerMap: removing container" podUID="5dc6676c-58c3-4ad9-9701-e76f70622bb8" containerName="istio-proxy" Nov 03 15:39:00.136767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136646 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc6676c-58c3-4ad9-9701-e76f70622bb8" containerName="istio-proxy" Nov 03 15:39:00.136767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136653 4557 cpu_manager.go:414] "containerMap: removing container" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" Nov 03 15:39:00.136767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136657 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" Nov 03 15:39:00.136767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136734 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc6676c-58c3-4ad9-9701-e76f70622bb8" containerName="istio-proxy" Nov 03 15:39:00.136767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136741 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="b43d3a39-a80b-4a80-b04b-0ac762251667" containerName="main" Nov 03 15:39:00.136767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136748 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f82529-9942-46d4-98dc-fa872122b973" containerName="istio-proxy" Nov 03 15:39:00.136767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136755 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d54cd1-cc1f-488b-9702-739043f4753c" containerName="main" Nov 03 15:39:00.136767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136764 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d2a233-488a-42d3-8e7b-cd7868a6f167" containerName="istio-proxy" Nov 03 15:39:00.136767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.136771 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f7013a-125c-4b34-8f4b-b72f312179d8" containerName="istio-proxy" Nov 03 15:39:00.141476 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.141458 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64" Nov 03 15:39:00.195497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.195471 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:39:00.199033 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.199012 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64"] Nov 03 15:39:00.222735 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.222706 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsftf\" (UniqueName: \"kubernetes.io/projected/1d32fc59-2cb1-4da5-989f-a4729dea933c-kube-api-access-nsftf\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64\" (UID: \"1d32fc59-2cb1-4da5-989f-a4729dea933c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64" Nov 03 15:39:00.222839 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.222761 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d32fc59-2cb1-4da5-989f-a4729dea933c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64\" (UID: \"1d32fc59-2cb1-4da5-989f-a4729dea933c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64" Nov 03 15:39:00.222947 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.222925 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d32fc59-2cb1-4da5-989f-a4729dea933c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64\" (UID: \"1d32fc59-2cb1-4da5-989f-a4729dea933c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64" Nov 03 15:39:00.245391 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.245365 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:39:00.286737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.286709 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d2a233-488a-42d3-8e7b-cd7868a6f167" path="/var/lib/kubelet/pods/d6d2a233-488a-42d3-8e7b-cd7868a6f167/volumes" Nov 03 15:39:00.324522 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.324490 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d32fc59-2cb1-4da5-989f-a4729dea933c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64\" (UID: \"1d32fc59-2cb1-4da5-989f-a4729dea933c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64" Nov 03 15:39:00.324676 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.324611 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsftf\" (UniqueName: \"kubernetes.io/projected/1d32fc59-2cb1-4da5-989f-a4729dea933c-kube-api-access-nsftf\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64\" (UID: \"1d32fc59-2cb1-4da5-989f-a4729dea933c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64" Nov 03 15:39:00.324676 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.324652 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d32fc59-2cb1-4da5-989f-a4729dea933c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64\" (UID: \"1d32fc59-2cb1-4da5-989f-a4729dea933c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64" Nov 03 15:39:00.325321 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.325301 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d32fc59-2cb1-4da5-989f-a4729dea933c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64\" (UID: \"1d32fc59-2cb1-4da5-989f-a4729dea933c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64" Nov 03 15:39:00.326842 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.326820 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d32fc59-2cb1-4da5-989f-a4729dea933c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64\" (UID: \"1d32fc59-2cb1-4da5-989f-a4729dea933c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64" Nov 03 15:39:00.352545 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.352514 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:00.386843 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.386772 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsftf\" (UniqueName: \"kubernetes.io/projected/1d32fc59-2cb1-4da5-989f-a4729dea933c-kube-api-access-nsftf\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64\" (UID: \"1d32fc59-2cb1-4da5-989f-a4729dea933c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64" Nov 03 15:39:00.454683 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.454656 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64" Nov 03 15:39:00.835842 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:00.835806 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64"] Nov 03 15:39:00.837393 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:39:00.837367 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d32fc59_2cb1_4da5_989f_a4729dea933c.slice/crio-d8d74d59b41da06e2f67997630e258be91c6937bb0017ac35baf7964f2851eec WatchSource:0}: Error finding container d8d74d59b41da06e2f67997630e258be91c6937bb0017ac35baf7964f2851eec: Status 404 returned error can't find the container with id d8d74d59b41da06e2f67997630e258be91c6937bb0017ac35baf7964f2851eec Nov 03 15:39:01.054865 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:01.054825 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64" event={"ID":"1d32fc59-2cb1-4da5-989f-a4729dea933c","Type":"ContainerStarted","Data":"d8d74d59b41da06e2f67997630e258be91c6937bb0017ac35baf7964f2851eec"} Nov 03 15:39:02.062441 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:02.062403 4557 generic.go:356] "Generic (PLEG): container finished" podID="1d32fc59-2cb1-4da5-989f-a4729dea933c" containerID="05c3df4df09339a9600e716e03364b9739214e63efa0503418636d7c1c5b9e9f" exitCode=0 Nov 03 15:39:02.062904 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:02.062483 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64" event={"ID":"1d32fc59-2cb1-4da5-989f-a4729dea933c","Type":"ContainerDied","Data":"05c3df4df09339a9600e716e03364b9739214e63efa0503418636d7c1c5b9e9f"} Nov 03 15:39:02.342945 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:02.342901 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:02.586030 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:02.585997 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:02.772208 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:02.772113 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:03.218827 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:03.218797 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64" Nov 03 15:39:03.274070 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:03.274038 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d32fc59-2cb1-4da5-989f-a4729dea933c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d32fc59-2cb1-4da5-989f-a4729dea933c-maas-api-openshift-ca\") pod \"1d32fc59-2cb1-4da5-989f-a4729dea933c\" (UID: \"1d32fc59-2cb1-4da5-989f-a4729dea933c\") " Nov 03 15:39:03.274282 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:03.274143 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d32fc59-2cb1-4da5-989f-a4729dea933c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d32fc59-2cb1-4da5-989f-a4729dea933c-maas-api-xks-ca\") pod \"1d32fc59-2cb1-4da5-989f-a4729dea933c\" (UID: \"1d32fc59-2cb1-4da5-989f-a4729dea933c\") " Nov 03 15:39:03.274366 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:03.274295 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d32fc59-2cb1-4da5-989f-a4729dea933c-kube-api-access-nsftf\" (UniqueName: \"kubernetes.io/projected/1d32fc59-2cb1-4da5-989f-a4729dea933c-kube-api-access-nsftf\") pod \"1d32fc59-2cb1-4da5-989f-a4729dea933c\" (UID: \"1d32fc59-2cb1-4da5-989f-a4729dea933c\") " Nov 03 15:39:03.274496 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:03.274472 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d32fc59-2cb1-4da5-989f-a4729dea933c-maas-api-openshift-ca" pod "1d32fc59-2cb1-4da5-989f-a4729dea933c" (UID: "1d32fc59-2cb1-4da5-989f-a4729dea933c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:39:03.274735 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:03.274715 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d32fc59-2cb1-4da5-989f-a4729dea933c-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:39:03.276231 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:03.276205 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d32fc59-2cb1-4da5-989f-a4729dea933c-maas-api-xks-ca" pod "1d32fc59-2cb1-4da5-989f-a4729dea933c" (UID: "1d32fc59-2cb1-4da5-989f-a4729dea933c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:39:03.276352 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:03.276269 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d32fc59-2cb1-4da5-989f-a4729dea933c-kube-api-access-nsftf" pod "1d32fc59-2cb1-4da5-989f-a4729dea933c" (UID: "1d32fc59-2cb1-4da5-989f-a4729dea933c"). InnerVolumeSpecName "kube-api-access-nsftf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:39:03.376446 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:03.376349 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d32fc59-2cb1-4da5-989f-a4729dea933c-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:39:03.376446 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:03.376391 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsftf\" (UniqueName: \"kubernetes.io/projected/1d32fc59-2cb1-4da5-989f-a4729dea933c-kube-api-access-nsftf\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:39:04.076173 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:04.076150 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64" Nov 03 15:39:04.076346 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:04.076146 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64" event={"ID":"1d32fc59-2cb1-4da5-989f-a4729dea933c","Type":"ContainerDied","Data":"d8d74d59b41da06e2f67997630e258be91c6937bb0017ac35baf7964f2851eec"} Nov 03 15:39:04.076346 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:04.076276 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8d74d59b41da06e2f67997630e258be91c6937bb0017ac35baf7964f2851eec" Nov 03 15:39:05.707191 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:05.707157 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:06.132292 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:06.132260 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:06.540859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:06.540770 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:08.930771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:08.930736 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:09.125201 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:09.125166 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:10.902243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:10.902176 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:12.068431 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:12.068368 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:12.276637 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:12.276528 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:13.079691 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:13.079658 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:13.303418 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:13.303388 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:15.698903 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:15.698863 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv"] Nov 03 15:39:15.699369 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:15.699356 4557 cpu_manager.go:414] "containerMap: removing container" podUID="1d32fc59-2cb1-4da5-989f-a4729dea933c" containerName="revoke-keys" Nov 03 15:39:15.699419 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:15.699371 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d32fc59-2cb1-4da5-989f-a4729dea933c" containerName="revoke-keys" Nov 03 15:39:15.699476 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:15.699467 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d32fc59-2cb1-4da5-989f-a4729dea933c" containerName="revoke-keys" Nov 03 15:39:15.703008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:15.702986 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv" Nov 03 15:39:15.763312 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:15.763285 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv"] Nov 03 15:39:15.809808 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:15.809783 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:39:15.809960 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:15.809872 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:39:15.818415 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:15.818392 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a91a7b7b-27b0-485d-9da4-8b19173918ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e2f4a7b-shkwv\" (UID: \"a91a7b7b-27b0-485d-9da4-8b19173918ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv" Nov 03 15:39:15.818483 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:15.818444 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a91a7b7b-27b0-485d-9da4-8b19173918ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e2f4a7b-shkwv\" (UID: \"a91a7b7b-27b0-485d-9da4-8b19173918ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv" Nov 03 15:39:15.818518 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:15.818500 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwkkg\" (UniqueName: \"kubernetes.io/projected/a91a7b7b-27b0-485d-9da4-8b19173918ca-kube-api-access-nwkkg\") pod \"maas-api-revoke-sub-5e2f4a7b-shkwv\" (UID: \"a91a7b7b-27b0-485d-9da4-8b19173918ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv" Nov 03 15:39:15.919800 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:15.919771 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a91a7b7b-27b0-485d-9da4-8b19173918ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e2f4a7b-shkwv\" (UID: \"a91a7b7b-27b0-485d-9da4-8b19173918ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv" Nov 03 15:39:15.920004 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:15.919812 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwkkg\" (UniqueName: \"kubernetes.io/projected/a91a7b7b-27b0-485d-9da4-8b19173918ca-kube-api-access-nwkkg\") pod \"maas-api-revoke-sub-5e2f4a7b-shkwv\" (UID: \"a91a7b7b-27b0-485d-9da4-8b19173918ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv" Nov 03 15:39:15.920004 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:15.919984 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a91a7b7b-27b0-485d-9da4-8b19173918ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e2f4a7b-shkwv\" (UID: \"a91a7b7b-27b0-485d-9da4-8b19173918ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv" Nov 03 15:39:15.920656 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:15.920633 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a91a7b7b-27b0-485d-9da4-8b19173918ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e2f4a7b-shkwv\" (UID: \"a91a7b7b-27b0-485d-9da4-8b19173918ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv" Nov 03 15:39:15.922235 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:15.922204 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a91a7b7b-27b0-485d-9da4-8b19173918ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e2f4a7b-shkwv\" (UID: \"a91a7b7b-27b0-485d-9da4-8b19173918ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv" Nov 03 15:39:15.983476 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:15.983403 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwkkg\" (UniqueName: \"kubernetes.io/projected/a91a7b7b-27b0-485d-9da4-8b19173918ca-kube-api-access-nwkkg\") pod \"maas-api-revoke-sub-5e2f4a7b-shkwv\" (UID: \"a91a7b7b-27b0-485d-9da4-8b19173918ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv" Nov 03 15:39:16.017661 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.017636 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv" Nov 03 15:39:16.217244 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.217207 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl"] Nov 03 15:39:16.221230 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.221213 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:39:16.274150 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.274074 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-f356bb-serving-cert\"" Nov 03 15:39:16.286780 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.286753 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl"] Nov 03 15:39:16.324064 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.324037 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkcmg\" (UniqueName: \"kubernetes.io/projected/76677f41-4a57-4d11-afb4-6fc9ec4c8299-kube-api-access-vkcmg\") pod \"maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl\" (UID: \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:39:16.324217 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.324094 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76677f41-4a57-4d11-afb4-6fc9ec4c8299-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl\" (UID: \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:39:16.324256 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.324234 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76677f41-4a57-4d11-afb4-6fc9ec4c8299-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl\" (UID: \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:39:16.324338 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.324326 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76677f41-4a57-4d11-afb4-6fc9ec4c8299-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl\" (UID: \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:39:16.393074 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.393035 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv"] Nov 03 15:39:16.394260 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:39:16.394223 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda91a7b7b_27b0_485d_9da4_8b19173918ca.slice/crio-0e20828f39598f74e4f6d962e0140de426f43bd258c5b7df5bcc08b9bb2e6e1f WatchSource:0}: Error finding container 0e20828f39598f74e4f6d962e0140de426f43bd258c5b7df5bcc08b9bb2e6e1f: Status 404 returned error can't find the container with id 0e20828f39598f74e4f6d962e0140de426f43bd258c5b7df5bcc08b9bb2e6e1f Nov 03 15:39:16.426006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.425982 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkcmg\" (UniqueName: \"kubernetes.io/projected/76677f41-4a57-4d11-afb4-6fc9ec4c8299-kube-api-access-vkcmg\") pod \"maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl\" (UID: \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:39:16.426096 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.426035 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76677f41-4a57-4d11-afb4-6fc9ec4c8299-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl\" (UID: \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:39:16.426096 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.426091 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76677f41-4a57-4d11-afb4-6fc9ec4c8299-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl\" (UID: \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:39:16.426210 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.426145 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76677f41-4a57-4d11-afb4-6fc9ec4c8299-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl\" (UID: \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:39:16.426267 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:39:16.426250 4557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-serving-cert: secret "maas-api-e2e-shared-b-w6-f356bb-serving-cert" not found Nov 03 15:39:16.426328 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:39:16.426306 4557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76677f41-4a57-4d11-afb4-6fc9ec4c8299-maas-api-tls podName:76677f41-4a57-4d11-afb4-6fc9ec4c8299 nodeName:}" failed. No retries permitted until 2026-11-03 15:39:16.926291924 +0000 UTC m=+1866.769413461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/76677f41-4a57-4d11-afb4-6fc9ec4c8299-maas-api-tls") pod "maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" (UID: "76677f41-4a57-4d11-afb4-6fc9ec4c8299") : secret "maas-api-e2e-shared-b-w6-f356bb-serving-cert" not found Nov 03 15:39:16.426756 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.426734 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76677f41-4a57-4d11-afb4-6fc9ec4c8299-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl\" (UID: \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:39:16.428283 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.428265 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76677f41-4a57-4d11-afb4-6fc9ec4c8299-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl\" (UID: \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:39:16.484420 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.484395 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:16.487705 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.487686 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkcmg\" (UniqueName: \"kubernetes.io/projected/76677f41-4a57-4d11-afb4-6fc9ec4c8299-kube-api-access-vkcmg\") pod \"maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl\" (UID: \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:39:16.529994 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:39:16.529926 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:39:16.553764 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.553738 4557 scope.go:122] "RemoveContainer" containerID="138d32feb911ebb6697ff62b15caa67006157d780b26727b01971faa9c49632e" Nov 03 15:39:16.563017 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.563000 4557 scope.go:122] "RemoveContainer" containerID="523594088ea5349b09daa86892727238d2b30212314a3779fc658c3663c753bf" Nov 03 15:39:16.571244 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.571228 4557 scope.go:122] "RemoveContainer" containerID="967125a7ab4054954fe4017938510f3ad637912912db8ec2573dd3b19cb475ba" Nov 03 15:39:16.583098 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.583080 4557 scope.go:122] "RemoveContainer" containerID="74c5e60261a37f086340b97aca19b1267528ce0afbc1d981eeb1930416ac3f1c" Nov 03 15:39:16.593724 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.593706 4557 scope.go:122] "RemoveContainer" containerID="518cbc3bbf7cfa353ed9052a4cecc8b372371420861615edcee99ce3c1cd702d" Nov 03 15:39:16.696394 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.696365 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:16.933374 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.933340 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76677f41-4a57-4d11-afb4-6fc9ec4c8299-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl\" (UID: \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:39:16.935814 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:16.935793 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76677f41-4a57-4d11-afb4-6fc9ec4c8299-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl\" (UID: \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:39:17.135173 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:17.135132 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:39:17.186000 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:17.185909 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv" event={"ID":"a91a7b7b-27b0-485d-9da4-8b19173918ca","Type":"ContainerStarted","Data":"f8880f52e865663f697df9ccd5d3b5a3859b11b8c75a24497ef2dd829434e589"} Nov 03 15:39:17.186000 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:17.185973 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv" event={"ID":"a91a7b7b-27b0-485d-9da4-8b19173918ca","Type":"ContainerStarted","Data":"0e20828f39598f74e4f6d962e0140de426f43bd258c5b7df5bcc08b9bb2e6e1f"} Nov 03 15:39:17.311135 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:17.311079 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv" podStartSLOduration=2.311061488 podStartE2EDuration="2.311061488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:39:17.30647418 +0000 UTC m=+1867.149595722" watchObservedRunningTime="2026-11-03 15:39:17.311061488 +0000 UTC m=+1867.154183031" Nov 03 15:39:17.515647 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:17.515619 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl"] Nov 03 15:39:17.517195 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:39:17.517169 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76677f41_4a57_4d11_afb4_6fc9ec4c8299.slice/crio-af4faf7594c490ca3b809e5aad8c0a5ef1dcf783620af0319c9fa1a090383fd9 WatchSource:0}: Error finding container af4faf7594c490ca3b809e5aad8c0a5ef1dcf783620af0319c9fa1a090383fd9: Status 404 returned error can't find the container with id af4faf7594c490ca3b809e5aad8c0a5ef1dcf783620af0319c9fa1a090383fd9 Nov 03 15:39:18.193721 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:18.193685 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" event={"ID":"76677f41-4a57-4d11-afb4-6fc9ec4c8299","Type":"ContainerStarted","Data":"af4faf7594c490ca3b809e5aad8c0a5ef1dcf783620af0319c9fa1a090383fd9"} Nov 03 15:39:18.195378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:18.195351 4557 generic.go:356] "Generic (PLEG): container finished" podID="a91a7b7b-27b0-485d-9da4-8b19173918ca" containerID="f8880f52e865663f697df9ccd5d3b5a3859b11b8c75a24497ef2dd829434e589" exitCode=0 Nov 03 15:39:18.195517 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:18.195416 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv" event={"ID":"a91a7b7b-27b0-485d-9da4-8b19173918ca","Type":"ContainerDied","Data":"f8880f52e865663f697df9ccd5d3b5a3859b11b8c75a24497ef2dd829434e589"} Nov 03 15:39:19.203400 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:19.203364 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" event={"ID":"76677f41-4a57-4d11-afb4-6fc9ec4c8299","Type":"ContainerStarted","Data":"86259576fe788fded66fd63e19b17a38a78e5167da652d83fa1e5764092fefcb"} Nov 03 15:39:19.329340 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:19.329275 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" podStartSLOduration=2.465472984 podStartE2EDuration="3.329256785s" totalImagesPullingTime="863.783801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:39:19.32588041 +0000 UTC m=+1869.169001951" watchObservedRunningTime="2026-11-03 15:39:19.329256785 +0000 UTC m=+1869.172378326" Nov 03 15:39:19.349812 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:19.349782 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv" Nov 03 15:39:19.372383 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:19.372352 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a91a7b7b-27b0-485d-9da4-8b19173918ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a91a7b7b-27b0-485d-9da4-8b19173918ca-maas-api-xks-ca\") pod \"a91a7b7b-27b0-485d-9da4-8b19173918ca\" (UID: \"a91a7b7b-27b0-485d-9da4-8b19173918ca\") " Nov 03 15:39:19.372507 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:19.372402 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a91a7b7b-27b0-485d-9da4-8b19173918ca-kube-api-access-nwkkg\" (UniqueName: \"kubernetes.io/projected/a91a7b7b-27b0-485d-9da4-8b19173918ca-kube-api-access-nwkkg\") pod \"a91a7b7b-27b0-485d-9da4-8b19173918ca\" (UID: \"a91a7b7b-27b0-485d-9da4-8b19173918ca\") " Nov 03 15:39:19.372507 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:19.372460 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a91a7b7b-27b0-485d-9da4-8b19173918ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a91a7b7b-27b0-485d-9da4-8b19173918ca-maas-api-openshift-ca\") pod \"a91a7b7b-27b0-485d-9da4-8b19173918ca\" (UID: \"a91a7b7b-27b0-485d-9da4-8b19173918ca\") " Nov 03 15:39:19.372838 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:19.372817 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a91a7b7b-27b0-485d-9da4-8b19173918ca-maas-api-openshift-ca" pod "a91a7b7b-27b0-485d-9da4-8b19173918ca" (UID: "a91a7b7b-27b0-485d-9da4-8b19173918ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:39:19.374515 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:19.374490 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a91a7b7b-27b0-485d-9da4-8b19173918ca-maas-api-xks-ca" pod "a91a7b7b-27b0-485d-9da4-8b19173918ca" (UID: "a91a7b7b-27b0-485d-9da4-8b19173918ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:39:19.374515 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:19.374495 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91a7b7b-27b0-485d-9da4-8b19173918ca-kube-api-access-nwkkg" pod "a91a7b7b-27b0-485d-9da4-8b19173918ca" (UID: "a91a7b7b-27b0-485d-9da4-8b19173918ca"). InnerVolumeSpecName "kube-api-access-nwkkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:39:19.474339 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:19.474254 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwkkg\" (UniqueName: \"kubernetes.io/projected/a91a7b7b-27b0-485d-9da4-8b19173918ca-kube-api-access-nwkkg\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:39:19.474339 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:19.474294 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a91a7b7b-27b0-485d-9da4-8b19173918ca-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:39:19.474339 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:19.474313 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a91a7b7b-27b0-485d-9da4-8b19173918ca-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:39:19.646642 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:19.646609 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:20.211776 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:20.211748 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv" Nov 03 15:39:20.212222 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:20.211779 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv" event={"ID":"a91a7b7b-27b0-485d-9da4-8b19173918ca","Type":"ContainerDied","Data":"0e20828f39598f74e4f6d962e0140de426f43bd258c5b7df5bcc08b9bb2e6e1f"} Nov 03 15:39:20.212222 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:20.211839 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e20828f39598f74e4f6d962e0140de426f43bd258c5b7df5bcc08b9bb2e6e1f" Nov 03 15:39:20.212222 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:20.212050 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:39:20.295275 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:20.295244 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:20.502466 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:20.502388 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:21.925695 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:21.924321 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:26.223433 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:26.223402 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:39:26.903723 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:26.903689 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs"] Nov 03 15:39:26.904286 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:26.904267 4557 cpu_manager.go:414] "containerMap: removing container" podUID="a91a7b7b-27b0-485d-9da4-8b19173918ca" containerName="revoke-keys" Nov 03 15:39:26.904395 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:26.904287 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a91a7b7b-27b0-485d-9da4-8b19173918ca" containerName="revoke-keys" Nov 03 15:39:26.904479 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:26.904465 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="a91a7b7b-27b0-485d-9da4-8b19173918ca" containerName="revoke-keys" Nov 03 15:39:26.908281 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:26.908260 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:39:26.959370 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:26.959344 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0aa695-fb42-42d7-b979-55ffed84e5f1-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs\" (UID: \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:39:26.959455 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:26.959385 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72ggw\" (UniqueName: \"kubernetes.io/projected/aa0aa695-fb42-42d7-b979-55ffed84e5f1-kube-api-access-72ggw\") pod \"maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs\" (UID: \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:39:26.959629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:26.959605 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0aa695-fb42-42d7-b979-55ffed84e5f1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs\" (UID: \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:39:26.959692 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:26.959667 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0aa695-fb42-42d7-b979-55ffed84e5f1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs\" (UID: \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:39:27.013514 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:27.013489 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-771ea0-serving-cert\"" Nov 03 15:39:27.016435 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:27.016411 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs"] Nov 03 15:39:27.061126 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:27.061086 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0aa695-fb42-42d7-b979-55ffed84e5f1-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs\" (UID: \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:39:27.062236 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:27.061165 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72ggw\" (UniqueName: \"kubernetes.io/projected/aa0aa695-fb42-42d7-b979-55ffed84e5f1-kube-api-access-72ggw\") pod \"maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs\" (UID: \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:39:27.062236 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:27.061338 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0aa695-fb42-42d7-b979-55ffed84e5f1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs\" (UID: \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:39:27.062236 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:27.061385 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0aa695-fb42-42d7-b979-55ffed84e5f1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs\" (UID: \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:39:27.062236 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:39:27.061511 4557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-serving-cert: secret "maas-api-e2e-shared-a-w1-771ea0-serving-cert" not found Nov 03 15:39:27.062236 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:39:27.061572 4557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa0aa695-fb42-42d7-b979-55ffed84e5f1-maas-api-tls podName:aa0aa695-fb42-42d7-b979-55ffed84e5f1 nodeName:}" failed. No retries permitted until 2026-11-03 15:39:27.561554384 +0000 UTC m=+1877.404675919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa0aa695-fb42-42d7-b979-55ffed84e5f1-maas-api-tls") pod "maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" (UID: "aa0aa695-fb42-42d7-b979-55ffed84e5f1") : secret "maas-api-e2e-shared-a-w1-771ea0-serving-cert" not found Nov 03 15:39:27.062236 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:27.062127 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0aa695-fb42-42d7-b979-55ffed84e5f1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs\" (UID: \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:39:27.063932 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:27.063890 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0aa695-fb42-42d7-b979-55ffed84e5f1-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs\" (UID: \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:39:27.122612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:27.122574 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72ggw\" (UniqueName: \"kubernetes.io/projected/aa0aa695-fb42-42d7-b979-55ffed84e5f1-kube-api-access-72ggw\") pod \"maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs\" (UID: \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:39:27.165316 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:27.165189 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:27.356878 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:27.356817 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:27.543936 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:27.543896 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:27.567756 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:27.567729 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0aa695-fb42-42d7-b979-55ffed84e5f1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs\" (UID: \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:39:27.570120 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:27.570094 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0aa695-fb42-42d7-b979-55ffed84e5f1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs\" (UID: \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:39:27.821789 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:27.821705 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:39:28.200560 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:28.200527 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs"] Nov 03 15:39:28.203651 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:39:28.203621 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0aa695_fb42_42d7_b979_55ffed84e5f1.slice/crio-218d1ca06f7e3fbc9a144bbf25b42cc775f97c42645cc0c074cc3006db89d6cf WatchSource:0}: Error finding container 218d1ca06f7e3fbc9a144bbf25b42cc775f97c42645cc0c074cc3006db89d6cf: Status 404 returned error can't find the container with id 218d1ca06f7e3fbc9a144bbf25b42cc775f97c42645cc0c074cc3006db89d6cf Nov 03 15:39:28.263173 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:28.263135 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" event={"ID":"aa0aa695-fb42-42d7-b979-55ffed84e5f1","Type":"ContainerStarted","Data":"218d1ca06f7e3fbc9a144bbf25b42cc775f97c42645cc0c074cc3006db89d6cf"} Nov 03 15:39:29.276792 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:29.276695 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" event={"ID":"aa0aa695-fb42-42d7-b979-55ffed84e5f1","Type":"ContainerStarted","Data":"06127e62cc58605130fb9e951cdda1e80a8a1fa874f90e61851487850aba09c8"} Nov 03 15:39:29.277286 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:29.276792 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:39:29.397284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:29.397230 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" podStartSLOduration=2.61321242 podStartE2EDuration="3.397217202s" totalImagesPullingTime="784.004782ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:39:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:39:29.396207134 +0000 UTC m=+1879.239328686" watchObservedRunningTime="2026-11-03 15:39:29.397217202 +0000 UTC m=+1879.240338743" Nov 03 15:39:32.205477 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:32.205437 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:35.290405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:35.290374 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:39:37.085913 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:37.085875 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:38.950194 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:38.950159 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:40.808893 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:40.808854 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:41.997922 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:41.997876 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:45.659919 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:45.659860 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:46.185647 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.185616 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:46.534757 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:39:46.534675 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:39:46.572807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.572772 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:46.789909 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.789828 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp"] Nov 03 15:39:46.794011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.793994 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.806526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.806493 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-credential-socket\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.806798 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.806767 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-envoy\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.806937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.806850 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-workload-certs\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.806937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.806930 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-podinfo\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.807058 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.806986 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmdlq\" (UniqueName: \"kubernetes.io/projected/3503bbfc-0896-44d5-8b21-02bfa7a360d0-kube-api-access-rmdlq\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.807122 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.807054 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-token\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.807122 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.807113 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-data\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.807229 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.807136 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istiod-ca-cert\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.807229 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.807163 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-workload-socket\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.848052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.848018 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-78034f-openshift-default-dockercfg-fjdbm\"" Nov 03 15:39:46.902993 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.902946 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp"] Nov 03 15:39:46.908093 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.908050 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-data\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.908257 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.908104 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istiod-ca-cert\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.908257 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.908134 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-workload-socket\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.908257 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.908200 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-credential-socket\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.908257 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.908252 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-envoy\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.908486 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.908308 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-workload-certs\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.908486 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.908376 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-podinfo\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.908486 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.908425 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmdlq\" (UniqueName: \"kubernetes.io/projected/3503bbfc-0896-44d5-8b21-02bfa7a360d0-kube-api-access-rmdlq\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.908678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.908490 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-token\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.912307 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.909290 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-data\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.912307 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.910506 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istiod-ca-cert\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.912307 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.910853 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-workload-socket\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.912307 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.911099 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-credential-socket\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.912307 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.911902 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-workload-certs\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.936604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.933122 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:46.946930 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.946901 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-podinfo\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:46.961516 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:46.961480 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-envoy\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:47.009049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:47.009015 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-token\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:47.020644 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:47.010471 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmdlq\" (UniqueName: \"kubernetes.io/projected/3503bbfc-0896-44d5-8b21-02bfa7a360d0-kube-api-access-rmdlq\") pod \"e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:47.112982 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:47.112939 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:47.377912 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:47.377825 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:47.561557 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:47.561522 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp"] Nov 03 15:39:47.563574 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:39:47.563551 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3503bbfc_0896_44d5_8b21_02bfa7a360d0.slice/crio-5f8c17ca17f905187796060ed816b85e2624d577fa6d49e8f32b8caee33a4ebe WatchSource:0}: Error finding container 5f8c17ca17f905187796060ed816b85e2624d577fa6d49e8f32b8caee33a4ebe: Status 404 returned error can't find the container with id 5f8c17ca17f905187796060ed816b85e2624d577fa6d49e8f32b8caee33a4ebe Nov 03 15:39:47.566763 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:47.566732 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:39:47.566860 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:47.566806 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:39:47.566922 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:47.566850 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:39:48.170954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:48.170919 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:48.387845 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:48.387795 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:48.388500 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:48.388471 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" event={"ID":"3503bbfc-0896-44d5-8b21-02bfa7a360d0","Type":"ContainerStarted","Data":"32b07a3fb046af46137f5f70e6ff769b295c9b215dda6d2710ba87886d76ec5d"} Nov 03 15:39:48.388635 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:48.388529 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" event={"ID":"3503bbfc-0896-44d5-8b21-02bfa7a360d0","Type":"ContainerStarted","Data":"5f8c17ca17f905187796060ed816b85e2624d577fa6d49e8f32b8caee33a4ebe"} Nov 03 15:39:48.560381 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:48.560337 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" podStartSLOduration=2.560323271 podStartE2EDuration="2.560323271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:39:48.509826945 +0000 UTC m=+1898.352948511" watchObservedRunningTime="2026-11-03 15:39:48.560323271 +0000 UTC m=+1898.403444812" Nov 03 15:39:48.585650 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:48.585617 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:48.787998 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:48.787956 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64"] Nov 03 15:39:48.792689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:48.792657 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-8a0e5a-4ln64"] Nov 03 15:39:49.114683 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:49.114652 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:49.120249 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:49.120222 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:49.294357 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:49.294325 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:49.394215 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:49.394144 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:49.395006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:49.394986 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:39:50.286203 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:50.286169 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d32fc59-2cb1-4da5-989f-a4729dea933c" path="/var/lib/kubelet/pods/1d32fc59-2cb1-4da5-989f-a4729dea933c/volumes" Nov 03 15:39:51.512224 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:51.512192 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:51.731463 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:51.731433 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:51.975498 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:51.975468 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:53.889717 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:53.889658 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:55.411856 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:55.411817 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:55.613367 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:55.613298 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:55.833605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:55.829034 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:56.911543 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:56.911506 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:57.098179 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:57.098142 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:58.157383 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:58.157342 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:58.902330 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:58.902255 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:59.244299 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:59.244208 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:59.546814 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:59.546763 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:39:59.821022 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:39:59.820933 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:04.472135 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:04.472070 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:04.666482 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:04.666450 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:04.862072 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:04.862037 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:06.470409 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:06.470373 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:07.763712 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:07.763680 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:07.957911 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:07.957867 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:09.334545 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:09.334504 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:09.527621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:09.527563 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:09.856147 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:09.856117 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:14.947475 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:14.947431 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:16.529873 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:40:16.529838 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:40:16.682999 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:16.682969 4557 scope.go:122] "RemoveContainer" containerID="28e783856b6154307b32d2e8206bb798143c819f24e0a2fc2b12f2ccbbb15c54" Nov 03 15:40:20.171662 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:20.171626 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:25.001629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:25.001525 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:26.909931 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:26.909896 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:27.099328 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:27.099297 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:29.282424 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:29.282388 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:30.824559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:30.824527 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:31.051329 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:31.051301 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:31.240359 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:31.240282 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:31.317914 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:31.317876 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8"] Nov 03 15:40:31.367485 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:31.367453 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f78fd6-pr5w8"] Nov 03 15:40:31.782245 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:31.782219 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:32.285604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:32.285550 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53ef9ece-80a8-44a3-9c92-1aa529ded720" path="/var/lib/kubelet/pods/53ef9ece-80a8-44a3-9c92-1aa529ded720/volumes" Nov 03 15:40:36.325839 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:36.325783 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:38.471635 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:38.468167 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:38.718719 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:38.718680 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:38.930082 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:38.930028 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:39.145469 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:39.145439 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:41.634689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:41.634645 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:43.089229 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:43.089195 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:46.530099 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:40:46.530063 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:40:47.020077 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:47.020042 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:47.629372 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:47.629334 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:48.070795 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:48.070758 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:48.700818 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:48.700725 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:49.027377 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:49.027294 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:52.446751 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:52.446711 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:53.229130 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:53.229090 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:53.459928 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:53.459899 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:53.692565 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:53.692529 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:54.664136 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:54.664102 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:54.921158 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:54.921073 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:56.653073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:56.653045 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:56.879314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:56.879248 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:40:58.148932 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:40:58.148896 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:03.200377 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:03.200335 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:06.839215 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:06.839184 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:08.043857 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:08.043817 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:08.230850 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:08.230817 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:08.931491 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:08.931455 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:09.121365 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:09.121327 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:12.679187 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:12.679149 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:12.998526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:12.998438 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:15.656993 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:15.656955 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:15.848960 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:15.848928 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:16.530681 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:41:16.530645 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:41:16.705329 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:16.705293 4557 scope.go:122] "RemoveContainer" containerID="0a44467e8abef767cca147f38eec0432eee4ea8cf5c440a3285aae001906e410" Nov 03 15:41:17.954234 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:17.954154 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:18.152563 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:18.152521 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:18.345488 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:18.345451 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:20.884110 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:20.884074 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:22.478407 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:22.478374 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8"] Nov 03 15:41:22.478849 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:22.478603 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" podUID="af721495-75e3-4b73-bb07-c291abae9b7c" containerName="maas-api" containerID="cri-o://b7c3d0908dea5bb8e80fd87424821f7dedb3853e28a44383367e55a772b4180d" gracePeriod=30 Nov 03 15:41:22.752779 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:22.752381 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:22.803270 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:22.803240 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:41:22.931514 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:22.931481 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af721495-75e3-4b73-bb07-c291abae9b7c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af721495-75e3-4b73-bb07-c291abae9b7c-maas-api-tls\") pod \"af721495-75e3-4b73-bb07-c291abae9b7c\" (UID: \"af721495-75e3-4b73-bb07-c291abae9b7c\") " Nov 03 15:41:22.931795 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:22.931533 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af721495-75e3-4b73-bb07-c291abae9b7c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af721495-75e3-4b73-bb07-c291abae9b7c-metrics-certs\") pod \"af721495-75e3-4b73-bb07-c291abae9b7c\" (UID: \"af721495-75e3-4b73-bb07-c291abae9b7c\") " Nov 03 15:41:22.931795 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:22.931623 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af721495-75e3-4b73-bb07-c291abae9b7c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af721495-75e3-4b73-bb07-c291abae9b7c-openshift-service-ca\") pod \"af721495-75e3-4b73-bb07-c291abae9b7c\" (UID: \"af721495-75e3-4b73-bb07-c291abae9b7c\") " Nov 03 15:41:22.931795 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:22.931740 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af721495-75e3-4b73-bb07-c291abae9b7c-kube-api-access-4njcr\" (UniqueName: \"kubernetes.io/projected/af721495-75e3-4b73-bb07-c291abae9b7c-kube-api-access-4njcr\") pod \"af721495-75e3-4b73-bb07-c291abae9b7c\" (UID: \"af721495-75e3-4b73-bb07-c291abae9b7c\") " Nov 03 15:41:22.932360 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:22.932305 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af721495-75e3-4b73-bb07-c291abae9b7c-openshift-service-ca" pod "af721495-75e3-4b73-bb07-c291abae9b7c" (UID: "af721495-75e3-4b73-bb07-c291abae9b7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:41:22.932521 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:22.932498 4557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af721495-75e3-4b73-bb07-c291abae9b7c-openshift-service-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:22.934515 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:22.934473 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af721495-75e3-4b73-bb07-c291abae9b7c-metrics-certs" pod "af721495-75e3-4b73-bb07-c291abae9b7c" (UID: "af721495-75e3-4b73-bb07-c291abae9b7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:41:22.936300 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:22.936274 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af721495-75e3-4b73-bb07-c291abae9b7c-kube-api-access-4njcr" pod "af721495-75e3-4b73-bb07-c291abae9b7c" (UID: "af721495-75e3-4b73-bb07-c291abae9b7c"). InnerVolumeSpecName "kube-api-access-4njcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:41:22.936611 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:22.936457 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af721495-75e3-4b73-bb07-c291abae9b7c-maas-api-tls" pod "af721495-75e3-4b73-bb07-c291abae9b7c" (UID: "af721495-75e3-4b73-bb07-c291abae9b7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:41:23.033276 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:23.033202 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af721495-75e3-4b73-bb07-c291abae9b7c-maas-api-tls\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:23.033276 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:23.033230 4557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af721495-75e3-4b73-bb07-c291abae9b7c-metrics-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:23.033276 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:23.033241 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4njcr\" (UniqueName: \"kubernetes.io/projected/af721495-75e3-4b73-bb07-c291abae9b7c-kube-api-access-4njcr\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:23.055922 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:23.055881 4557 generic.go:356] "Generic (PLEG): container finished" podID="af721495-75e3-4b73-bb07-c291abae9b7c" containerID="b7c3d0908dea5bb8e80fd87424821f7dedb3853e28a44383367e55a772b4180d" exitCode=0 Nov 03 15:41:23.056127 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:23.055950 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" Nov 03 15:41:23.056127 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:23.055964 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" event={"ID":"af721495-75e3-4b73-bb07-c291abae9b7c","Type":"ContainerDied","Data":"b7c3d0908dea5bb8e80fd87424821f7dedb3853e28a44383367e55a772b4180d"} Nov 03 15:41:23.056127 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:23.056100 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8" event={"ID":"af721495-75e3-4b73-bb07-c291abae9b7c","Type":"ContainerDied","Data":"0cc4eb1f0205726f8ec54d67e6bdbec98860b5e4c3ed3289b3ec7bfdcef5d73b"} Nov 03 15:41:23.056288 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:23.056136 4557 scope.go:122] "RemoveContainer" containerID="b7c3d0908dea5bb8e80fd87424821f7dedb3853e28a44383367e55a772b4180d" Nov 03 15:41:23.083175 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:23.083151 4557 scope.go:122] "RemoveContainer" containerID="b7c3d0908dea5bb8e80fd87424821f7dedb3853e28a44383367e55a772b4180d" Nov 03 15:41:23.083836 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:41:23.083783 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7c3d0908dea5bb8e80fd87424821f7dedb3853e28a44383367e55a772b4180d\": container with ID starting with b7c3d0908dea5bb8e80fd87424821f7dedb3853e28a44383367e55a772b4180d not found: ID does not exist" containerID="b7c3d0908dea5bb8e80fd87424821f7dedb3853e28a44383367e55a772b4180d" Nov 03 15:41:23.083949 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:23.083840 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7c3d0908dea5bb8e80fd87424821f7dedb3853e28a44383367e55a772b4180d"} err="failed to get container status \"b7c3d0908dea5bb8e80fd87424821f7dedb3853e28a44383367e55a772b4180d\": rpc error: code = NotFound desc = could not find container \"b7c3d0908dea5bb8e80fd87424821f7dedb3853e28a44383367e55a772b4180d\": container with ID starting with b7c3d0908dea5bb8e80fd87424821f7dedb3853e28a44383367e55a772b4180d not found: ID does not exist" Nov 03 15:41:23.224947 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:23.224904 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8"] Nov 03 15:41:23.229730 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:23.229707 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-7b3fd3-b84bf7fb5-lqhq8"] Nov 03 15:41:24.231349 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:24.231318 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:24.285738 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:24.285708 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af721495-75e3-4b73-bb07-c291abae9b7c" path="/var/lib/kubelet/pods/af721495-75e3-4b73-bb07-c291abae9b7c/volumes" Nov 03 15:41:24.464321 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:24.464290 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:24.705083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:24.705048 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:25.207373 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:25.207320 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:29.369529 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:29.369493 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:29.581593 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:29.581545 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:29.811480 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:29.811453 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:30.070219 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:30.070140 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd"] Nov 03 15:41:30.119770 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:30.119741 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ae485a-kfntd"] Nov 03 15:41:30.286158 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:30.286129 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7596d6db-b29f-4686-ada3-d58864dcc8c1" path="/var/lib/kubelet/pods/7596d6db-b29f-4686-ada3-d58864dcc8c1/volumes" Nov 03 15:41:31.843050 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:31.843017 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:32.062318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:32.062285 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:32.260535 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:32.260459 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:32.784048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:32.784012 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:35.132355 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:35.132302 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:37.714706 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:37.714673 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg"] Nov 03 15:41:37.764057 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:37.764024 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59"] Nov 03 15:41:37.857031 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:37.856994 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:40.533365 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:40.533281 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:45.708761 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:45.708722 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:45.976016 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:45.975941 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:46.531178 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:41:46.531133 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:41:50.857967 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:50.857919 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:41:52.015605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:52.015337 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:41:52.053606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:52.042706 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 15:41:52.164842 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:52.164069 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:41:52.471752 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:52.471705 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:41:52.472397 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:52.472355 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:41:52.545152 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:52.544746 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:41:52.736111 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:52.735388 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" containerID="cri-o://929f4994753e8be356377a8f8bf7ae1fb4d03088934901b86decbd0f0f7ba976" gracePeriod=45 Nov 03 15:41:52.792620 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:52.791978 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" containerID="cri-o://3dc73bccfbddbab4f758277cbb351920d7640a9e47d8a5cc57857bd6f29d70c3" gracePeriod=45 Nov 03 15:41:52.956605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:52.956538 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": dial tcp 10.133.0.38:8000: connect: connection refused" Nov 03 15:41:53.127063 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.127027 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:41:53.143376 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.143346 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:41:53.216144 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.216111 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-tmp-dir\") pod \"0da69fa9-d199-4563-9c71-6627369d0ef9\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " Nov 03 15:41:53.216311 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.216185 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-dshm\") pod \"0da69fa9-d199-4563-9c71-6627369d0ef9\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " Nov 03 15:41:53.216311 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.216210 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-model-cache\") pod \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " Nov 03 15:41:53.216311 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.216274 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-home\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-home\") pod \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " Nov 03 15:41:53.216453 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.216336 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-tmp-dir\") pod \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " Nov 03 15:41:53.216453 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.216381 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-home\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-home\") pod \"0da69fa9-d199-4563-9c71-6627369d0ef9\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " Nov 03 15:41:53.216453 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.216435 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-tls-certs\") pod \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " Nov 03 15:41:53.216653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.216485 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-kube-api-access-skjld\" (UniqueName: \"kubernetes.io/projected/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-kube-api-access-skjld\") pod \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " Nov 03 15:41:53.216653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.216545 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-model-cache\") pod \"0da69fa9-d199-4563-9c71-6627369d0ef9\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " Nov 03 15:41:53.216653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.216433 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-tmp-dir" pod "0da69fa9-d199-4563-9c71-6627369d0ef9" (UID: "0da69fa9-d199-4563-9c71-6627369d0ef9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:53.216653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.216571 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-home" pod "2003121d-2d9e-41ae-a8e5-239b5af7b8d7" (UID: "2003121d-2d9e-41ae-a8e5-239b5af7b8d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:53.216653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.216558 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-model-cache" pod "2003121d-2d9e-41ae-a8e5-239b5af7b8d7" (UID: "2003121d-2d9e-41ae-a8e5-239b5af7b8d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:53.216653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.216631 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da69fa9-d199-4563-9c71-6627369d0ef9-tls-certs\" (UniqueName: \"kubernetes.io/secret/0da69fa9-d199-4563-9c71-6627369d0ef9-tls-certs\") pod \"0da69fa9-d199-4563-9c71-6627369d0ef9\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " Nov 03 15:41:53.217064 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.216668 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-home" pod "0da69fa9-d199-4563-9c71-6627369d0ef9" (UID: "0da69fa9-d199-4563-9c71-6627369d0ef9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:53.217064 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.216727 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da69fa9-d199-4563-9c71-6627369d0ef9-kube-api-access-8htmp\" (UniqueName: \"kubernetes.io/projected/0da69fa9-d199-4563-9c71-6627369d0ef9-kube-api-access-8htmp\") pod \"0da69fa9-d199-4563-9c71-6627369d0ef9\" (UID: \"0da69fa9-d199-4563-9c71-6627369d0ef9\") " Nov 03 15:41:53.217064 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.216720 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-tmp-dir" pod "2003121d-2d9e-41ae-a8e5-239b5af7b8d7" (UID: "2003121d-2d9e-41ae-a8e5-239b5af7b8d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:53.217064 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.216821 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-model-cache" pod "0da69fa9-d199-4563-9c71-6627369d0ef9" (UID: "0da69fa9-d199-4563-9c71-6627369d0ef9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:53.217064 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.216859 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-dshm\") pod \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\" (UID: \"2003121d-2d9e-41ae-a8e5-239b5af7b8d7\") " Nov 03 15:41:53.217482 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.217449 4557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-tmp-dir\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.217573 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.217488 4557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-model-cache\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.217573 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.217503 4557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-home\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.217573 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.217518 4557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-tmp-dir\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.217573 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.217568 4557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-home\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.217834 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.217603 4557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-model-cache\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.219064 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.219029 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-dshm" pod "2003121d-2d9e-41ae-a8e5-239b5af7b8d7" (UID: "2003121d-2d9e-41ae-a8e5-239b5af7b8d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:53.219064 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.219053 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da69fa9-d199-4563-9c71-6627369d0ef9-tls-certs" pod "0da69fa9-d199-4563-9c71-6627369d0ef9" (UID: "0da69fa9-d199-4563-9c71-6627369d0ef9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:41:53.219213 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.219104 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-dshm" pod "0da69fa9-d199-4563-9c71-6627369d0ef9" (UID: "0da69fa9-d199-4563-9c71-6627369d0ef9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:41:53.219213 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.219167 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-kube-api-access-skjld" pod "2003121d-2d9e-41ae-a8e5-239b5af7b8d7" (UID: "2003121d-2d9e-41ae-a8e5-239b5af7b8d7"). InnerVolumeSpecName "kube-api-access-skjld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:41:53.219279 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.219228 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-tls-certs" pod "2003121d-2d9e-41ae-a8e5-239b5af7b8d7" (UID: "2003121d-2d9e-41ae-a8e5-239b5af7b8d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:41:53.219388 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.219368 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da69fa9-d199-4563-9c71-6627369d0ef9-kube-api-access-8htmp" pod "0da69fa9-d199-4563-9c71-6627369d0ef9" (UID: "0da69fa9-d199-4563-9c71-6627369d0ef9"). InnerVolumeSpecName "kube-api-access-8htmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:41:53.327626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.319098 4557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da69fa9-d199-4563-9c71-6627369d0ef9-dshm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.327626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.319132 4557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-tls-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.327626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.319149 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skjld\" (UniqueName: \"kubernetes.io/projected/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-kube-api-access-skjld\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.327626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.319165 4557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da69fa9-d199-4563-9c71-6627369d0ef9-tls-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.327626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.319181 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8htmp\" (UniqueName: \"kubernetes.io/projected/0da69fa9-d199-4563-9c71-6627369d0ef9-kube-api-access-8htmp\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.327626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.319199 4557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2003121d-2d9e-41ae-a8e5-239b5af7b8d7-dshm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:41:53.338143 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.338109 4557 generic.go:356] "Generic (PLEG): container finished" podID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerID="3dc73bccfbddbab4f758277cbb351920d7640a9e47d8a5cc57857bd6f29d70c3" exitCode=0 Nov 03 15:41:53.338328 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.338166 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" Nov 03 15:41:53.338328 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.338202 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" event={"ID":"0da69fa9-d199-4563-9c71-6627369d0ef9","Type":"ContainerDied","Data":"3dc73bccfbddbab4f758277cbb351920d7640a9e47d8a5cc57857bd6f29d70c3"} Nov 03 15:41:53.338328 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.338258 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59" event={"ID":"0da69fa9-d199-4563-9c71-6627369d0ef9","Type":"ContainerDied","Data":"457b1fd2fc6d13b5436df070b36ab6c38fa9db8910cd97fe74002a5a3643fa38"} Nov 03 15:41:53.338328 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.338282 4557 scope.go:122] "RemoveContainer" containerID="3dc73bccfbddbab4f758277cbb351920d7640a9e47d8a5cc57857bd6f29d70c3" Nov 03 15:41:53.341046 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.341007 4557 generic.go:356] "Generic (PLEG): container finished" podID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerID="929f4994753e8be356377a8f8bf7ae1fb4d03088934901b86decbd0f0f7ba976" exitCode=0 Nov 03 15:41:53.341185 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.341146 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" event={"ID":"2003121d-2d9e-41ae-a8e5-239b5af7b8d7","Type":"ContainerDied","Data":"929f4994753e8be356377a8f8bf7ae1fb4d03088934901b86decbd0f0f7ba976"} Nov 03 15:41:53.341185 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.341177 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" event={"ID":"2003121d-2d9e-41ae-a8e5-239b5af7b8d7","Type":"ContainerDied","Data":"f55498cbc68b61de3d0ab36f35d40ddab5442814ff9b7965201b08edd6da4641"} Nov 03 15:41:53.341292 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.341238 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg" Nov 03 15:41:53.351626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.349902 4557 scope.go:122] "RemoveContainer" containerID="3dc73bccfbddbab4f758277cbb351920d7640a9e47d8a5cc57857bd6f29d70c3" Nov 03 15:41:53.351626 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:41:53.350710 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc73bccfbddbab4f758277cbb351920d7640a9e47d8a5cc57857bd6f29d70c3\": container with ID starting with 3dc73bccfbddbab4f758277cbb351920d7640a9e47d8a5cc57857bd6f29d70c3 not found: ID does not exist" containerID="3dc73bccfbddbab4f758277cbb351920d7640a9e47d8a5cc57857bd6f29d70c3" Nov 03 15:41:53.351626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.350746 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc73bccfbddbab4f758277cbb351920d7640a9e47d8a5cc57857bd6f29d70c3"} err="failed to get container status \"3dc73bccfbddbab4f758277cbb351920d7640a9e47d8a5cc57857bd6f29d70c3\": rpc error: code = NotFound desc = could not find container \"3dc73bccfbddbab4f758277cbb351920d7640a9e47d8a5cc57857bd6f29d70c3\": container with ID starting with 3dc73bccfbddbab4f758277cbb351920d7640a9e47d8a5cc57857bd6f29d70c3 not found: ID does not exist" Nov 03 15:41:53.351626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.350769 4557 scope.go:122] "RemoveContainer" containerID="929f4994753e8be356377a8f8bf7ae1fb4d03088934901b86decbd0f0f7ba976" Nov 03 15:41:53.366757 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.366733 4557 scope.go:122] "RemoveContainer" containerID="929f4994753e8be356377a8f8bf7ae1fb4d03088934901b86decbd0f0f7ba976" Nov 03 15:41:53.367133 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:41:53.367106 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929f4994753e8be356377a8f8bf7ae1fb4d03088934901b86decbd0f0f7ba976\": container with ID starting with 929f4994753e8be356377a8f8bf7ae1fb4d03088934901b86decbd0f0f7ba976 not found: ID does not exist" containerID="929f4994753e8be356377a8f8bf7ae1fb4d03088934901b86decbd0f0f7ba976" Nov 03 15:41:53.367248 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.367143 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929f4994753e8be356377a8f8bf7ae1fb4d03088934901b86decbd0f0f7ba976"} err="failed to get container status \"929f4994753e8be356377a8f8bf7ae1fb4d03088934901b86decbd0f0f7ba976\": rpc error: code = NotFound desc = could not find container \"929f4994753e8be356377a8f8bf7ae1fb4d03088934901b86decbd0f0f7ba976\": container with ID starting with 929f4994753e8be356377a8f8bf7ae1fb4d03088934901b86decbd0f0f7ba976 not found: ID does not exist" Nov 03 15:41:53.516605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.516401 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg"] Nov 03 15:41:53.522635 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.520518 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/premium-w5-34e8c5-kserve-5499d7c559-v27gg"] Nov 03 15:41:53.636554 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.636469 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59"] Nov 03 15:41:53.640359 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:53.640330 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-34e8c5/simulator-w5-34e8c5-kserve-7b45d84cbb-d4h59"] Nov 03 15:41:54.290559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:54.290513 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" path="/var/lib/kubelet/pods/0da69fa9-d199-4563-9c71-6627369d0ef9/volumes" Nov 03 15:41:54.291776 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:54.291751 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" path="/var/lib/kubelet/pods/2003121d-2d9e-41ae-a8e5-239b5af7b8d7/volumes" Nov 03 15:41:56.084464 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:41:56.084396 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:00.386411 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.386379 4557 generic.go:356] "Generic (PLEG): container finished" podID="3503bbfc-0896-44d5-8b21-02bfa7a360d0" containerID="32b07a3fb046af46137f5f70e6ff769b295c9b215dda6d2710ba87886d76ec5d" exitCode=0 Nov 03 15:42:00.386904 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.386457 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" event={"ID":"3503bbfc-0896-44d5-8b21-02bfa7a360d0","Type":"ContainerDied","Data":"32b07a3fb046af46137f5f70e6ff769b295c9b215dda6d2710ba87886d76ec5d"} Nov 03 15:42:00.386904 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.386832 4557 scope.go:122] "RemoveContainer" containerID="32b07a3fb046af46137f5f70e6ff769b295c9b215dda6d2710ba87886d76ec5d" Nov 03 15:42:00.388377 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.388350 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:42:00.388465 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.388414 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:42:00.388465 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.388448 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:42:00.582911 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.582880 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb"] Nov 03 15:42:00.583396 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.583382 4557 cpu_manager.go:414] "containerMap: removing container" podUID="af721495-75e3-4b73-bb07-c291abae9b7c" containerName="maas-api" Nov 03 15:42:00.583440 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.583399 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af721495-75e3-4b73-bb07-c291abae9b7c" containerName="maas-api" Nov 03 15:42:00.583440 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.583415 4557 cpu_manager.go:414] "containerMap: removing container" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" Nov 03 15:42:00.583440 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.583419 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" Nov 03 15:42:00.583440 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.583428 4557 cpu_manager.go:414] "containerMap: removing container" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" Nov 03 15:42:00.583440 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.583433 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" Nov 03 15:42:00.583613 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.583521 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da69fa9-d199-4563-9c71-6627369d0ef9" containerName="main" Nov 03 15:42:00.583613 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.583530 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="2003121d-2d9e-41ae-a8e5-239b5af7b8d7" containerName="main" Nov 03 15:42:00.583613 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.583541 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="af721495-75e3-4b73-bb07-c291abae9b7c" containerName="maas-api" Nov 03 15:42:00.586840 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.586816 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.698711 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.698626 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb"] Nov 03 15:42:00.713111 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.713075 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-90fcca-openshift-default-dockercfg-g7lcc\"" Nov 03 15:42:00.713286 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.713127 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.713286 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.713177 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.713425 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.713307 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acfc1375-2b70-4aec-b161-4e24c39a8d59-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.713484 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.713436 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.713548 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.713489 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.713548 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.713523 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.713682 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.713613 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.713764 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.713743 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.713830 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.713784 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87cv5\" (UniqueName: \"kubernetes.io/projected/acfc1375-2b70-4aec-b161-4e24c39a8d59-kube-api-access-87cv5\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.815376 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.815338 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.815376 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.815375 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.815625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.815393 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.815625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.815451 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.815625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.815475 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87cv5\" (UniqueName: \"kubernetes.io/projected/acfc1375-2b70-4aec-b161-4e24c39a8d59-kube-api-access-87cv5\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.815841 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.815810 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.815946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.815686 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.816054 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.815928 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.816054 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.816016 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.816156 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.816097 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acfc1375-2b70-4aec-b161-4e24c39a8d59-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.816213 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.816198 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.816270 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.816252 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.816612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.816555 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.816766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.816748 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acfc1375-2b70-4aec-b161-4e24c39a8d59-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.817762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.817739 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.818930 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.818908 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.877506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.877465 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87cv5\" (UniqueName: \"kubernetes.io/projected/acfc1375-2b70-4aec-b161-4e24c39a8d59-kube-api-access-87cv5\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.878516 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.878483 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-mslrb\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.903546 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.903521 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:00.964493 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:00.964370 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:01.351899 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:01.351872 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb"] Nov 03 15:42:01.353092 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:42:01.353063 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfc1375_2b70_4aec_b161_4e24c39a8d59.slice/crio-16e299be3d2ca7a71d4e20ec264c0531456681ade8ac49edb6cc63a85298931b WatchSource:0}: Error finding container 16e299be3d2ca7a71d4e20ec264c0531456681ade8ac49edb6cc63a85298931b: Status 404 returned error can't find the container with id 16e299be3d2ca7a71d4e20ec264c0531456681ade8ac49edb6cc63a85298931b Nov 03 15:42:01.355721 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:01.355689 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:42:01.355843 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:01.355753 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:42:01.355843 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:01.355784 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:42:01.395229 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:01.395199 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" event={"ID":"3503bbfc-0896-44d5-8b21-02bfa7a360d0","Type":"ContainerStarted","Data":"2d60679a1424b6925ab6fafb959a6a7eef64509bbc56c11d06667be52b1a3382"} Nov 03 15:42:01.396688 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:01.396662 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" event={"ID":"acfc1375-2b70-4aec-b161-4e24c39a8d59","Type":"ContainerStarted","Data":"16e299be3d2ca7a71d4e20ec264c0531456681ade8ac49edb6cc63a85298931b"} Nov 03 15:42:02.115391 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:02.115352 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:42:02.115848 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:02.115812 4557 patch_prober.go:29] interesting pod/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 15:42:02.116027 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:02.116002 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" podUID="3503bbfc-0896-44d5-8b21-02bfa7a360d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 15:42:02.431668 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:02.431295 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" event={"ID":"acfc1375-2b70-4aec-b161-4e24c39a8d59","Type":"ContainerStarted","Data":"15cc961e3e3f64b185896db9d129d4d27f921b1c188aa480feaa7ec359b0ac2c"} Nov 03 15:42:02.562730 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:02.562672 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" podStartSLOduration=2.5626597049999997 podStartE2EDuration="2.562659705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:42:02.560718112 +0000 UTC m=+2032.403839653" watchObservedRunningTime="2026-11-03 15:42:02.562659705 +0000 UTC m=+2032.405781246" Nov 03 15:42:02.921605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:02.919750 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:02.926610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:02.925782 4557 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 15:42:02.926610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:02.925859 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" podUID="acfc1375-2b70-4aec-b161-4e24c39a8d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 15:42:03.122606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:03.119029 4557 patch_prober.go:29] interesting pod/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 15:42:03.122606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:03.119096 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" podUID="3503bbfc-0896-44d5-8b21-02bfa7a360d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 15:42:03.904615 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:03.904031 4557 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 15:42:03.904615 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:03.904117 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" podUID="acfc1375-2b70-4aec-b161-4e24c39a8d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 15:42:04.121606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:04.113993 4557 patch_prober.go:29] interesting pod/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 15:42:04.121606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:04.114054 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" podUID="3503bbfc-0896-44d5-8b21-02bfa7a360d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 15:42:04.399604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:04.398690 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:42:04.907605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:04.903981 4557 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 15:42:04.907605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:04.904046 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" podUID="acfc1375-2b70-4aec-b161-4e24c39a8d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 15:42:05.126606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:05.125983 4557 patch_prober.go:29] interesting pod/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 15:42:05.126606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:05.126065 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" podUID="3503bbfc-0896-44d5-8b21-02bfa7a360d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 15:42:05.933606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:05.932717 4557 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 15:42:05.933606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:05.932795 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" podUID="acfc1375-2b70-4aec-b161-4e24c39a8d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 15:42:05.972006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:05.971971 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:06.116606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:06.113972 4557 patch_prober.go:29] interesting pod/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 15:42:06.116606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:06.114033 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" podUID="3503bbfc-0896-44d5-8b21-02bfa7a360d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 15:42:06.175980 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:06.175945 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:06.457105 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:06.457071 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:06.911603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:06.910976 4557 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 15:42:06.911603 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:06.911037 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" podUID="acfc1375-2b70-4aec-b161-4e24c39a8d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 15:42:07.114853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:07.114820 4557 patch_prober.go:29] interesting pod/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 15:42:07.115268 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:07.114878 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" podUID="3503bbfc-0896-44d5-8b21-02bfa7a360d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 15:42:07.262008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:07.261930 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:07.649756 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:07.649725 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:07.910663 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:07.910564 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:07.911108 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:07.911086 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:07.912019 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:07.911999 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:42:08.119494 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:08.119462 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:42:08.120763 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:08.120744 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:42:10.604953 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:10.604919 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:11.404443 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:11.404410 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:15.249705 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:15.249647 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:15.552180 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:15.552143 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:16.495796 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:16.495753 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:16.531409 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:42:16.531373 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:42:18.683862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:18.683820 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:21.626034 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:21.625997 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:25.253667 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:25.253617 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:25.507810 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:25.507725 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:26.498525 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:26.498485 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:31.530933 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:31.530880 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:35.749284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:35.749249 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:36.805194 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:36.805121 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:37.238816 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:37.238674 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:37.473598 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:37.473550 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:41.349671 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:41.349571 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:43.231370 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:43.231327 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:43.444343 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:43.444311 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:43.652397 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:43.652366 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:43.872755 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:43.872720 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:46.011285 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.011183 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp"] Nov 03 15:42:46.015696 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.015679 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.060800 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.060751 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd"] Nov 03 15:42:46.065322 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.065299 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.147307 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.147271 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.147482 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.147324 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.147482 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.147398 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.147482 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.147465 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g24pm\" (UniqueName: \"kubernetes.io/projected/2a6c2b03-8f2a-47cd-8e66-923505d511d7-kube-api-access-g24pm\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.147669 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.147535 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d78dz\" (UniqueName: \"kubernetes.io/projected/327437b8-3b27-4490-9cf6-d965799a1c3a-kube-api-access-d78dz\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.147669 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.147563 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.147758 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.147674 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.147758 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.147716 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.147758 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.147740 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.147927 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.147856 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/327437b8-3b27-4490-9cf6-d965799a1c3a-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.147969 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.147922 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.148007 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.147980 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.148041 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.148015 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.148041 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.148031 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.148144 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.148129 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.148184 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.148176 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.148223 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.148192 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.148264 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.148224 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.157398 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.157370 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp"] Nov 03 15:42:46.160819 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.160793 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd"] Nov 03 15:42:46.249118 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249086 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.249303 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249133 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.249303 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249177 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.249303 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249198 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g24pm\" (UniqueName: \"kubernetes.io/projected/2a6c2b03-8f2a-47cd-8e66-923505d511d7-kube-api-access-g24pm\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.249303 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249216 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d78dz\" (UniqueName: \"kubernetes.io/projected/327437b8-3b27-4490-9cf6-d965799a1c3a-kube-api-access-d78dz\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.249303 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249237 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.249303 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249284 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.249689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249320 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.249689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249345 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.249689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249391 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/327437b8-3b27-4490-9cf6-d965799a1c3a-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.249689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249442 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.249689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249489 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.249689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249528 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.249689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249555 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.249689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249605 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.249689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249649 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.249689 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249686 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.250220 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249715 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.250220 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249739 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.250220 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249764 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.250220 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.249935 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.250220 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.250082 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.250220 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.250175 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.251424 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.251005 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.251424 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.251379 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.251736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.251560 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/327437b8-3b27-4490-9cf6-d965799a1c3a-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.265278 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.252792 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.265278 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.253194 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.265278 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.254441 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.265278 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.254851 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.265612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.265561 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.266514 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.266486 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.311410 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.311384 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.311563 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.311427 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g24pm\" (UniqueName: \"kubernetes.io/projected/2a6c2b03-8f2a-47cd-8e66-923505d511d7-kube-api-access-g24pm\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.311563 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.311485 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.312943 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.312922 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d78dz\" (UniqueName: \"kubernetes.io/projected/327437b8-3b27-4490-9cf6-d965799a1c3a-kube-api-access-d78dz\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.328642 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.328623 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:46.378713 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.378682 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:46.533509 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:42:46.533419 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:42:46.795970 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.795889 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp"] Nov 03 15:42:46.799443 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:42:46.799334 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6c2b03_8f2a_47cd_8e66_923505d511d7.slice/crio-0329d7eb4e67010e91205d6e476a346034a9233398ebc3cc1897104e263a0de1 WatchSource:0}: Error finding container 0329d7eb4e67010e91205d6e476a346034a9233398ebc3cc1897104e263a0de1: Status 404 returned error can't find the container with id 0329d7eb4e67010e91205d6e476a346034a9233398ebc3cc1897104e263a0de1 Nov 03 15:42:46.801193 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.801169 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd"] Nov 03 15:42:46.802430 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:42:46.802408 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327437b8_3b27_4490_9cf6_d965799a1c3a.slice/crio-b47a85b3a7ebb294d769bb4df22f2c8758d07ba2b19204c25fcf5a2b6f9d7b67 WatchSource:0}: Error finding container b47a85b3a7ebb294d769bb4df22f2c8758d07ba2b19204c25fcf5a2b6f9d7b67: Status 404 returned error can't find the container with id b47a85b3a7ebb294d769bb4df22f2c8758d07ba2b19204c25fcf5a2b6f9d7b67 Nov 03 15:42:46.802530 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.802488 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:42:46.802593 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.802567 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:42:46.802646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.802630 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:42:46.805154 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.805127 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:42:46.805236 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.805184 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:42:46.805236 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:46.805210 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:42:47.738137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:47.738102 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" event={"ID":"2a6c2b03-8f2a-47cd-8e66-923505d511d7","Type":"ContainerStarted","Data":"c0eccdc48adfeedf19b06d2e30b748a2704e88b8c55d8b15df46fdf90dcf3c9d"} Nov 03 15:42:47.738762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:47.738173 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" event={"ID":"2a6c2b03-8f2a-47cd-8e66-923505d511d7","Type":"ContainerStarted","Data":"0329d7eb4e67010e91205d6e476a346034a9233398ebc3cc1897104e263a0de1"} Nov 03 15:42:47.740168 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:47.740128 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" event={"ID":"327437b8-3b27-4490-9cf6-d965799a1c3a","Type":"ContainerStarted","Data":"6b7f1e2478ece1d7bc4c704c57719f10fc9f762c68a735aa04c0b1e7e7f5606d"} Nov 03 15:42:47.740300 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:47.740182 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" event={"ID":"327437b8-3b27-4490-9cf6-d965799a1c3a","Type":"ContainerStarted","Data":"b47a85b3a7ebb294d769bb4df22f2c8758d07ba2b19204c25fcf5a2b6f9d7b67"} Nov 03 15:42:47.930769 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:47.930700 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" podStartSLOduration=2.930678298 podStartE2EDuration="2.930678298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:42:47.918195901 +0000 UTC m=+2077.761317444" watchObservedRunningTime="2026-11-03 15:42:47.930678298 +0000 UTC m=+2077.773799841" Nov 03 15:42:48.040862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:48.040758 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" podStartSLOduration=3.04074217 podStartE2EDuration="3.04074217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:42:48.039338717 +0000 UTC m=+2077.882460260" watchObservedRunningTime="2026-11-03 15:42:48.04074217 +0000 UTC m=+2077.883863711" Nov 03 15:42:48.296688 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:48.294254 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:48.332656 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:48.332366 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:48.333455 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:48.333427 4557 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 15:42:48.333703 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:48.333674 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" podUID="2a6c2b03-8f2a-47cd-8e66-923505d511d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 15:42:48.385605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:48.382618 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:48.385605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:48.382732 4557 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 03 15:42:48.385605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:48.382770 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" podUID="327437b8-3b27-4490-9cf6-d965799a1c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 03 15:42:48.519606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:48.518673 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:48.827631 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:48.826108 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:49.330606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:49.329962 4557 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 15:42:49.330606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:49.330014 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" podUID="2a6c2b03-8f2a-47cd-8e66-923505d511d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 15:42:49.384606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:49.380921 4557 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 03 15:42:49.384606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:49.380980 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" podUID="327437b8-3b27-4490-9cf6-d965799a1c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 03 15:42:50.329605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:50.329129 4557 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 15:42:50.329605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:50.329203 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" podUID="2a6c2b03-8f2a-47cd-8e66-923505d511d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 15:42:50.399605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:50.399499 4557 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 03 15:42:50.399605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:50.399562 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" podUID="327437b8-3b27-4490-9cf6-d965799a1c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 03 15:42:51.329931 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:51.329898 4557 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 15:42:51.330345 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:51.329949 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" podUID="2a6c2b03-8f2a-47cd-8e66-923505d511d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 15:42:51.399605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:51.397357 4557 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 03 15:42:51.399605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:51.397408 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" podUID="327437b8-3b27-4490-9cf6-d965799a1c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 03 15:42:52.180607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:52.176950 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:52.329013 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:52.328981 4557 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 15:42:52.329176 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:52.329033 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" podUID="2a6c2b03-8f2a-47cd-8e66-923505d511d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 15:42:52.384605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:52.380920 4557 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 03 15:42:52.384605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:52.380979 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" podUID="327437b8-3b27-4490-9cf6-d965799a1c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 03 15:42:52.428732 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:52.428693 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:53.333344 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:53.333314 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:53.333799 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:53.333781 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:53.334403 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:53.334385 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:42:53.384194 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:53.384170 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:53.384509 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:53.384491 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:53.385165 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:53.385145 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:42:54.419619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:54.419563 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:54.638889 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:54.638836 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:54.841362 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:54.841332 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:58.030955 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:58.030923 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw"] Nov 03 15:42:58.143629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:58.143574 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95"] Nov 03 15:42:58.447125 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:58.447081 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:58.746688 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:58.746557 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:42:59.737913 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:42:59.737849 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:01.485668 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:01.485623 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:03.032123 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.032062 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw" podUID="67217abc-0f32-4031-ac20-5ef4293cc64e" containerName="payload-processing" containerID="cri-o://43b68841eb317bc830d335887ef9a4f8e077b11ca9483509cc17af21598765b4" gracePeriod=25 Nov 03 15:43:03.144289 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.144257 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95" podUID="795e739f-2842-4d1b-bbd7-5770099cc7bb" containerName="payload-pre-processing" containerID="cri-o://951eb282aefd9acbd7bae89b875a779078950e837f984e8162c916429e229789" gracePeriod=25 Nov 03 15:43:03.292358 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.292332 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw" Nov 03 15:43:03.336307 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.336275 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67217abc-0f32-4031-ac20-5ef4293cc64e-kube-api-access-b792s\" (UniqueName: \"kubernetes.io/projected/67217abc-0f32-4031-ac20-5ef4293cc64e-kube-api-access-b792s\") pod \"67217abc-0f32-4031-ac20-5ef4293cc64e\" (UID: \"67217abc-0f32-4031-ac20-5ef4293cc64e\") " Nov 03 15:43:03.336486 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.336335 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67217abc-0f32-4031-ac20-5ef4293cc64e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67217abc-0f32-4031-ac20-5ef4293cc64e-plugins-config-volume\") pod \"67217abc-0f32-4031-ac20-5ef4293cc64e\" (UID: \"67217abc-0f32-4031-ac20-5ef4293cc64e\") " Nov 03 15:43:03.336985 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.336952 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67217abc-0f32-4031-ac20-5ef4293cc64e-plugins-config-volume" pod "67217abc-0f32-4031-ac20-5ef4293cc64e" (UID: "67217abc-0f32-4031-ac20-5ef4293cc64e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:43:03.339554 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.339528 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67217abc-0f32-4031-ac20-5ef4293cc64e-kube-api-access-b792s" pod "67217abc-0f32-4031-ac20-5ef4293cc64e" (UID: "67217abc-0f32-4031-ac20-5ef4293cc64e"). InnerVolumeSpecName "kube-api-access-b792s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:43:03.402825 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.402791 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95" Nov 03 15:43:03.437762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.437733 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795e739f-2842-4d1b-bbd7-5770099cc7bb-kube-api-access-zxb5r\" (UniqueName: \"kubernetes.io/projected/795e739f-2842-4d1b-bbd7-5770099cc7bb-kube-api-access-zxb5r\") pod \"795e739f-2842-4d1b-bbd7-5770099cc7bb\" (UID: \"795e739f-2842-4d1b-bbd7-5770099cc7bb\") " Nov 03 15:43:03.438006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.437992 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/795e739f-2842-4d1b-bbd7-5770099cc7bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795e739f-2842-4d1b-bbd7-5770099cc7bb-plugins-config-volume\") pod \"795e739f-2842-4d1b-bbd7-5770099cc7bb\" (UID: \"795e739f-2842-4d1b-bbd7-5770099cc7bb\") " Nov 03 15:43:03.438408 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.438383 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b792s\" (UniqueName: \"kubernetes.io/projected/67217abc-0f32-4031-ac20-5ef4293cc64e-kube-api-access-b792s\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:03.438519 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.438413 4557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67217abc-0f32-4031-ac20-5ef4293cc64e-plugins-config-volume\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:03.438519 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.438389 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795e739f-2842-4d1b-bbd7-5770099cc7bb-plugins-config-volume" pod "795e739f-2842-4d1b-bbd7-5770099cc7bb" (UID: "795e739f-2842-4d1b-bbd7-5770099cc7bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:43:03.440481 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.440452 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795e739f-2842-4d1b-bbd7-5770099cc7bb-kube-api-access-zxb5r" pod "795e739f-2842-4d1b-bbd7-5770099cc7bb" (UID: "795e739f-2842-4d1b-bbd7-5770099cc7bb"). InnerVolumeSpecName "kube-api-access-zxb5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:43:03.539490 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.539407 4557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795e739f-2842-4d1b-bbd7-5770099cc7bb-plugins-config-volume\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:03.539490 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.539436 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxb5r\" (UniqueName: \"kubernetes.io/projected/795e739f-2842-4d1b-bbd7-5770099cc7bb-kube-api-access-zxb5r\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:03.907344 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.907313 4557 generic.go:356] "Generic (PLEG): container finished" podID="67217abc-0f32-4031-ac20-5ef4293cc64e" containerID="43b68841eb317bc830d335887ef9a4f8e077b11ca9483509cc17af21598765b4" exitCode=0 Nov 03 15:43:03.907555 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.907357 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw" Nov 03 15:43:03.907555 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.907403 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw" event={"ID":"67217abc-0f32-4031-ac20-5ef4293cc64e","Type":"ContainerDied","Data":"43b68841eb317bc830d335887ef9a4f8e077b11ca9483509cc17af21598765b4"} Nov 03 15:43:03.907555 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.907466 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw" event={"ID":"67217abc-0f32-4031-ac20-5ef4293cc64e","Type":"ContainerDied","Data":"625d3e567676f0dff46d97479bad5ccf9556284d026dd426f136eb0c51ab19ee"} Nov 03 15:43:03.907555 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.907498 4557 scope.go:122] "RemoveContainer" containerID="43b68841eb317bc830d335887ef9a4f8e077b11ca9483509cc17af21598765b4" Nov 03 15:43:03.909343 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.909319 4557 generic.go:356] "Generic (PLEG): container finished" podID="795e739f-2842-4d1b-bbd7-5770099cc7bb" containerID="951eb282aefd9acbd7bae89b875a779078950e837f984e8162c916429e229789" exitCode=0 Nov 03 15:43:03.909466 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.909361 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95" Nov 03 15:43:03.909466 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.909404 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95" event={"ID":"795e739f-2842-4d1b-bbd7-5770099cc7bb","Type":"ContainerDied","Data":"951eb282aefd9acbd7bae89b875a779078950e837f984e8162c916429e229789"} Nov 03 15:43:03.909466 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.909456 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95" event={"ID":"795e739f-2842-4d1b-bbd7-5770099cc7bb","Type":"ContainerDied","Data":"817fec5ad3ee9b7377da7097460a668d55b8516c47472500ae7a16e9c2ab2ec2"} Nov 03 15:43:03.918628 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.918519 4557 scope.go:122] "RemoveContainer" containerID="43b68841eb317bc830d335887ef9a4f8e077b11ca9483509cc17af21598765b4" Nov 03 15:43:03.918914 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:43:03.918891 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b68841eb317bc830d335887ef9a4f8e077b11ca9483509cc17af21598765b4\": container with ID starting with 43b68841eb317bc830d335887ef9a4f8e077b11ca9483509cc17af21598765b4 not found: ID does not exist" containerID="43b68841eb317bc830d335887ef9a4f8e077b11ca9483509cc17af21598765b4" Nov 03 15:43:03.919006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.918920 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b68841eb317bc830d335887ef9a4f8e077b11ca9483509cc17af21598765b4"} err="failed to get container status \"43b68841eb317bc830d335887ef9a4f8e077b11ca9483509cc17af21598765b4\": rpc error: code = NotFound desc = could not find container \"43b68841eb317bc830d335887ef9a4f8e077b11ca9483509cc17af21598765b4\": container with ID starting with 43b68841eb317bc830d335887ef9a4f8e077b11ca9483509cc17af21598765b4 not found: ID does not exist" Nov 03 15:43:03.919006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.918938 4557 scope.go:122] "RemoveContainer" containerID="951eb282aefd9acbd7bae89b875a779078950e837f984e8162c916429e229789" Nov 03 15:43:03.927891 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.927875 4557 scope.go:122] "RemoveContainer" containerID="951eb282aefd9acbd7bae89b875a779078950e837f984e8162c916429e229789" Nov 03 15:43:03.928156 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:43:03.928137 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"951eb282aefd9acbd7bae89b875a779078950e837f984e8162c916429e229789\": container with ID starting with 951eb282aefd9acbd7bae89b875a779078950e837f984e8162c916429e229789 not found: ID does not exist" containerID="951eb282aefd9acbd7bae89b875a779078950e837f984e8162c916429e229789" Nov 03 15:43:03.928203 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:03.928163 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"951eb282aefd9acbd7bae89b875a779078950e837f984e8162c916429e229789"} err="failed to get container status \"951eb282aefd9acbd7bae89b875a779078950e837f984e8162c916429e229789\": rpc error: code = NotFound desc = could not find container \"951eb282aefd9acbd7bae89b875a779078950e837f984e8162c916429e229789\": container with ID starting with 951eb282aefd9acbd7bae89b875a779078950e837f984e8162c916429e229789 not found: ID does not exist" Nov 03 15:43:04.078971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:04.078662 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw"] Nov 03 15:43:04.090628 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:04.090561 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5d3cde-6b4d46c887-j97gw"] Nov 03 15:43:04.204644 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:04.204511 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95"] Nov 03 15:43:04.208309 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:04.208266 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5d3cde-d67b6f6c5-t2j95"] Nov 03 15:43:04.286277 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:04.286239 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67217abc-0f32-4031-ac20-5ef4293cc64e" path="/var/lib/kubelet/pods/67217abc-0f32-4031-ac20-5ef4293cc64e/volumes" Nov 03 15:43:04.286683 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:04.286666 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="795e739f-2842-4d1b-bbd7-5770099cc7bb" path="/var/lib/kubelet/pods/795e739f-2842-4d1b-bbd7-5770099cc7bb/volumes" Nov 03 15:43:04.749410 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:04.749330 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:07.655462 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:07.655426 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:09.966403 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:09.966370 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:10.222610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:10.222510 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:10.559466 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:10.559441 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:43:10.567319 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:10.567293 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:43:10.726829 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:10.726784 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:10.956769 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:10.956645 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:15.008633 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:15.008524 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:16.530448 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:43:16.530415 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:43:16.753024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:16.752996 4557 scope.go:122] "RemoveContainer" containerID="9ebe4eb678040c576f1636f877a843c44368b708b9f046e25da0ed2f9ca4ae53" Nov 03 15:43:16.863797 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:16.863766 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:20.034838 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:20.034800 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:25.010487 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:25.010385 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:27.053730 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:27.053693 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:30.708804 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:30.708756 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:36.142142 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:36.142101 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:38.534072 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:38.534038 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm"] Nov 03 15:43:38.534479 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:38.534242 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" podUID="93d60ef8-6b7d-45af-8b6b-5579fb6ce960" containerName="istio-proxy" containerID="cri-o://8df1bf69ec9ad974fe9f2269a137969dd02ccd9f948bd3d6dbbffc5e30d826dc" gracePeriod=30 Nov 03 15:43:39.039305 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:39.039270 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm"] Nov 03 15:43:39.039505 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:39.039480 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" podUID="72be94c1-dd49-4f71-8a37-6056e0939ee9" containerName="istio-proxy" containerID="cri-o://8badc86a51cc816f5ac6db0185a2bb8f37c11cbe934663cfc33513b64d01fe91" gracePeriod=30 Nov 03 15:43:39.366102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:39.366068 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:43:39.366276 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:39.366127 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" podUID="93d60ef8-6b7d-45af-8b6b-5579fb6ce960" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:43:39.580337 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:39.580302 4557 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:43:39.580722 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:39.580366 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" podUID="72be94c1-dd49-4f71-8a37-6056e0939ee9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:43:39.936286 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:39.936249 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc"] Nov 03 15:43:40.039322 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:40.039289 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr"] Nov 03 15:43:40.335062 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:40.335029 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:40.579456 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:40.579422 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:41.432309 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:41.432271 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:43.804232 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.804177 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:43:43.863643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.863610 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-data\") pod \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " Nov 03 15:43:43.863831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.863654 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-podinfo\") pod \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " Nov 03 15:43:43.863831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.863703 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-envoy\") pod \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " Nov 03 15:43:43.863831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.863724 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-kube-api-access-7snj7\" (UniqueName: \"kubernetes.io/projected/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-kube-api-access-7snj7\") pod \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " Nov 03 15:43:43.863831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.863766 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istiod-ca-cert\") pod \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " Nov 03 15:43:43.863831 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.863828 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-token\" (UniqueName: \"kubernetes.io/projected/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-token\") pod \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " Nov 03 15:43:43.864113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.863873 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-workload-socket\") pod \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " Nov 03 15:43:43.864113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.863917 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-workload-certs\") pod \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " Nov 03 15:43:43.864113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.863915 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-data" pod "93d60ef8-6b7d-45af-8b6b-5579fb6ce960" (UID: "93d60ef8-6b7d-45af-8b6b-5579fb6ce960"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:43.864113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.864003 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-credential-socket\") pod \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\" (UID: \"93d60ef8-6b7d-45af-8b6b-5579fb6ce960\") " Nov 03 15:43:43.864309 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.864224 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-workload-socket" pod "93d60ef8-6b7d-45af-8b6b-5579fb6ce960" (UID: "93d60ef8-6b7d-45af-8b6b-5579fb6ce960"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:43.864309 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.864239 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-workload-certs" pod "93d60ef8-6b7d-45af-8b6b-5579fb6ce960" (UID: "93d60ef8-6b7d-45af-8b6b-5579fb6ce960"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:43.864485 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.864466 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-credential-socket" pod "93d60ef8-6b7d-45af-8b6b-5579fb6ce960" (UID: "93d60ef8-6b7d-45af-8b6b-5579fb6ce960"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:43.864485 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.864404 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istiod-ca-cert" pod "93d60ef8-6b7d-45af-8b6b-5579fb6ce960" (UID: "93d60ef8-6b7d-45af-8b6b-5579fb6ce960"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:43:43.864737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.864717 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:43.864785 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.864758 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:43.864785 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.864776 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:43.864852 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.864792 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:43.864852 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.864806 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:43.866181 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.866151 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-podinfo" pod "93d60ef8-6b7d-45af-8b6b-5579fb6ce960" (UID: "93d60ef8-6b7d-45af-8b6b-5579fb6ce960"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:43:43.866423 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.866398 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-token" pod "93d60ef8-6b7d-45af-8b6b-5579fb6ce960" (UID: "93d60ef8-6b7d-45af-8b6b-5579fb6ce960"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:43:43.866488 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.866428 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-envoy" pod "93d60ef8-6b7d-45af-8b6b-5579fb6ce960" (UID: "93d60ef8-6b7d-45af-8b6b-5579fb6ce960"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:43.866488 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.866469 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-kube-api-access-7snj7" pod "93d60ef8-6b7d-45af-8b6b-5579fb6ce960" (UID: "93d60ef8-6b7d-45af-8b6b-5579fb6ce960"). InnerVolumeSpecName "kube-api-access-7snj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:43:43.966437 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.966406 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:43.966437 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.966430 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:43.966437 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.966442 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7snj7\" (UniqueName: \"kubernetes.io/projected/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-kube-api-access-7snj7\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:43.966749 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:43.966453 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d60ef8-6b7d-45af-8b6b-5579fb6ce960-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.172013 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.171983 4557 generic.go:356] "Generic (PLEG): container finished" podID="72be94c1-dd49-4f71-8a37-6056e0939ee9" containerID="8badc86a51cc816f5ac6db0185a2bb8f37c11cbe934663cfc33513b64d01fe91" exitCode=0 Nov 03 15:43:44.172160 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.172064 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" event={"ID":"72be94c1-dd49-4f71-8a37-6056e0939ee9","Type":"ContainerDied","Data":"8badc86a51cc816f5ac6db0185a2bb8f37c11cbe934663cfc33513b64d01fe91"} Nov 03 15:43:44.173562 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.173528 4557 generic.go:356] "Generic (PLEG): container finished" podID="93d60ef8-6b7d-45af-8b6b-5579fb6ce960" containerID="8df1bf69ec9ad974fe9f2269a137969dd02ccd9f948bd3d6dbbffc5e30d826dc" exitCode=0 Nov 03 15:43:44.173705 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.173605 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" event={"ID":"93d60ef8-6b7d-45af-8b6b-5579fb6ce960","Type":"ContainerDied","Data":"8df1bf69ec9ad974fe9f2269a137969dd02ccd9f948bd3d6dbbffc5e30d826dc"} Nov 03 15:43:44.173705 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.173645 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" Nov 03 15:43:44.173705 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.173665 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm" event={"ID":"93d60ef8-6b7d-45af-8b6b-5579fb6ce960","Type":"ContainerDied","Data":"553771b10cc76e4d78d7e04d8726ba4413c95d3289e110c11ef3a8e0b64b794c"} Nov 03 15:43:44.173705 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.173696 4557 scope.go:122] "RemoveContainer" containerID="8df1bf69ec9ad974fe9f2269a137969dd02ccd9f948bd3d6dbbffc5e30d826dc" Nov 03 15:43:44.184026 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.184007 4557 scope.go:122] "RemoveContainer" containerID="8df1bf69ec9ad974fe9f2269a137969dd02ccd9f948bd3d6dbbffc5e30d826dc" Nov 03 15:43:44.184270 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:43:44.184251 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df1bf69ec9ad974fe9f2269a137969dd02ccd9f948bd3d6dbbffc5e30d826dc\": container with ID starting with 8df1bf69ec9ad974fe9f2269a137969dd02ccd9f948bd3d6dbbffc5e30d826dc not found: ID does not exist" containerID="8df1bf69ec9ad974fe9f2269a137969dd02ccd9f948bd3d6dbbffc5e30d826dc" Nov 03 15:43:44.184347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.184276 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df1bf69ec9ad974fe9f2269a137969dd02ccd9f948bd3d6dbbffc5e30d826dc"} err="failed to get container status \"8df1bf69ec9ad974fe9f2269a137969dd02ccd9f948bd3d6dbbffc5e30d826dc\": rpc error: code = NotFound desc = could not find container \"8df1bf69ec9ad974fe9f2269a137969dd02ccd9f948bd3d6dbbffc5e30d826dc\": container with ID starting with 8df1bf69ec9ad974fe9f2269a137969dd02ccd9f948bd3d6dbbffc5e30d826dc not found: ID does not exist" Nov 03 15:43:44.290280 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.290254 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:43:44.342253 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.342185 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm"] Nov 03 15:43:44.346730 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.346710 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-g6twm"] Nov 03 15:43:44.370785 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.370745 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-data\") pod \"72be94c1-dd49-4f71-8a37-6056e0939ee9\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " Nov 03 15:43:44.370890 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.370861 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72be94c1-dd49-4f71-8a37-6056e0939ee9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72be94c1-dd49-4f71-8a37-6056e0939ee9-istiod-ca-cert\") pod \"72be94c1-dd49-4f71-8a37-6056e0939ee9\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " Nov 03 15:43:44.370936 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.370903 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-token\" (UniqueName: \"kubernetes.io/projected/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-token\") pod \"72be94c1-dd49-4f71-8a37-6056e0939ee9\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " Nov 03 15:43:44.370936 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.370929 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-podinfo\") pod \"72be94c1-dd49-4f71-8a37-6056e0939ee9\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " Nov 03 15:43:44.371016 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.370956 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-workload-certs\") pod \"72be94c1-dd49-4f71-8a37-6056e0939ee9\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " Nov 03 15:43:44.371016 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.370994 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-envoy\") pod \"72be94c1-dd49-4f71-8a37-6056e0939ee9\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " Nov 03 15:43:44.371094 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.371004 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-data" pod "72be94c1-dd49-4f71-8a37-6056e0939ee9" (UID: "72be94c1-dd49-4f71-8a37-6056e0939ee9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:44.371094 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.371067 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-workload-socket\") pod \"72be94c1-dd49-4f71-8a37-6056e0939ee9\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " Nov 03 15:43:44.371196 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.371108 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-credential-socket\") pod \"72be94c1-dd49-4f71-8a37-6056e0939ee9\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " Nov 03 15:43:44.371252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.371215 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72be94c1-dd49-4f71-8a37-6056e0939ee9-kube-api-access-w7dqx\" (UniqueName: \"kubernetes.io/projected/72be94c1-dd49-4f71-8a37-6056e0939ee9-kube-api-access-w7dqx\") pod \"72be94c1-dd49-4f71-8a37-6056e0939ee9\" (UID: \"72be94c1-dd49-4f71-8a37-6056e0939ee9\") " Nov 03 15:43:44.371252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.371234 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-workload-certs" pod "72be94c1-dd49-4f71-8a37-6056e0939ee9" (UID: "72be94c1-dd49-4f71-8a37-6056e0939ee9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:44.371364 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.371326 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-workload-socket" pod "72be94c1-dd49-4f71-8a37-6056e0939ee9" (UID: "72be94c1-dd49-4f71-8a37-6056e0939ee9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:44.371364 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.371350 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72be94c1-dd49-4f71-8a37-6056e0939ee9-istiod-ca-cert" pod "72be94c1-dd49-4f71-8a37-6056e0939ee9" (UID: "72be94c1-dd49-4f71-8a37-6056e0939ee9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:43:44.371485 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.371443 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-credential-socket" pod "72be94c1-dd49-4f71-8a37-6056e0939ee9" (UID: "72be94c1-dd49-4f71-8a37-6056e0939ee9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:44.371807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.371786 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.371918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.371812 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72be94c1-dd49-4f71-8a37-6056e0939ee9-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.371918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.371836 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.371918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.371850 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.371918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.371863 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.373117 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.373101 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-podinfo" pod "72be94c1-dd49-4f71-8a37-6056e0939ee9" (UID: "72be94c1-dd49-4f71-8a37-6056e0939ee9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:43:44.373183 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.373150 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-envoy" pod "72be94c1-dd49-4f71-8a37-6056e0939ee9" (UID: "72be94c1-dd49-4f71-8a37-6056e0939ee9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:43:44.373444 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.373418 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-token" pod "72be94c1-dd49-4f71-8a37-6056e0939ee9" (UID: "72be94c1-dd49-4f71-8a37-6056e0939ee9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:43:44.373559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.373488 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72be94c1-dd49-4f71-8a37-6056e0939ee9-kube-api-access-w7dqx" pod "72be94c1-dd49-4f71-8a37-6056e0939ee9" (UID: "72be94c1-dd49-4f71-8a37-6056e0939ee9"). InnerVolumeSpecName "kube-api-access-w7dqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:43:44.473626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.472791 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.473626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.472827 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.473626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.472842 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72be94c1-dd49-4f71-8a37-6056e0939ee9-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.473626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.472857 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7dqx\" (UniqueName: \"kubernetes.io/projected/72be94c1-dd49-4f71-8a37-6056e0939ee9-kube-api-access-w7dqx\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:44.936817 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:44.936777 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc" podUID="8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6" containerName="payload-processing" containerID="cri-o://eebfba88b6b008db2ce3c1f2bca8cd1624461ba96ab4a01f9bcfc5a273fe4c99" gracePeriod=25 Nov 03 15:43:45.040430 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.040396 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr" podUID="9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc" containerName="payload-pre-processing" containerID="cri-o://8993a109a82f6046f96ca18b47c7bfcf8d306e4cdd0db66327897f3300046f17" gracePeriod=25 Nov 03 15:43:45.183110 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.183077 4557 generic.go:356] "Generic (PLEG): container finished" podID="8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6" containerID="eebfba88b6b008db2ce3c1f2bca8cd1624461ba96ab4a01f9bcfc5a273fe4c99" exitCode=0 Nov 03 15:43:45.183272 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.183179 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc" event={"ID":"8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6","Type":"ContainerDied","Data":"eebfba88b6b008db2ce3c1f2bca8cd1624461ba96ab4a01f9bcfc5a273fe4c99"} Nov 03 15:43:45.185215 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.185046 4557 generic.go:356] "Generic (PLEG): container finished" podID="9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc" containerID="8993a109a82f6046f96ca18b47c7bfcf8d306e4cdd0db66327897f3300046f17" exitCode=0 Nov 03 15:43:45.185215 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.185074 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr" event={"ID":"9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc","Type":"ContainerDied","Data":"8993a109a82f6046f96ca18b47c7bfcf8d306e4cdd0db66327897f3300046f17"} Nov 03 15:43:45.190681 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.187340 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" Nov 03 15:43:45.190681 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.187520 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm" event={"ID":"72be94c1-dd49-4f71-8a37-6056e0939ee9","Type":"ContainerDied","Data":"36ba4cc55b2e60e219e24d7ecda9032e875ec5fbb86f8e13c785a108e7eb014c"} Nov 03 15:43:45.190681 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.187563 4557 scope.go:122] "RemoveContainer" containerID="8badc86a51cc816f5ac6db0185a2bb8f37c11cbe934663cfc33513b64d01fe91" Nov 03 15:43:45.218231 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.218206 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc" Nov 03 15:43:45.289041 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.289009 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6-kube-api-access-gqwlg\" (UniqueName: \"kubernetes.io/projected/8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6-kube-api-access-gqwlg\") pod \"8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6\" (UID: \"8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6\") " Nov 03 15:43:45.289193 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.289052 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6-plugins-config-volume\") pod \"8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6\" (UID: \"8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6\") " Nov 03 15:43:45.289417 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.289389 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6-plugins-config-volume" pod "8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6" (UID: "8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:43:45.289702 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.289682 4557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6-plugins-config-volume\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:45.291875 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.291853 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6-kube-api-access-gqwlg" pod "8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6" (UID: "8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6"). InnerVolumeSpecName "kube-api-access-gqwlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:43:45.321858 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.321834 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr" Nov 03 15:43:45.391254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.391223 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqwlg\" (UniqueName: \"kubernetes.io/projected/8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6-kube-api-access-gqwlg\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:45.486114 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.486035 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm"] Nov 03 15:43:45.492223 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.492201 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc-kube-api-access-qpm5d\" (UniqueName: \"kubernetes.io/projected/9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc-kube-api-access-qpm5d\") pod \"9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc\" (UID: \"9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc\") " Nov 03 15:43:45.492325 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.492280 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc-plugins-config-volume\") pod \"9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc\" (UID: \"9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc\") " Nov 03 15:43:45.492680 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.492654 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc-plugins-config-volume" pod "9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc" (UID: "9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:43:45.494243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.494222 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc-kube-api-access-qpm5d" pod "9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc" (UID: "9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc"). InnerVolumeSpecName "kube-api-access-qpm5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:43:45.535636 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.535611 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-rkcjm"] Nov 03 15:43:45.594013 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.593987 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpm5d\" (UniqueName: \"kubernetes.io/projected/9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc-kube-api-access-qpm5d\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:45.594013 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:45.594012 4557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc-plugins-config-volume\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:43:46.196890 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:46.196857 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc" Nov 03 15:43:46.197319 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:46.196859 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc" event={"ID":"8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6","Type":"ContainerDied","Data":"662f9bbe03dfdb89584c50b1f2de9b4fd2a9397a5db9c43ba3127796b499406e"} Nov 03 15:43:46.197319 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:46.197005 4557 scope.go:122] "RemoveContainer" containerID="eebfba88b6b008db2ce3c1f2bca8cd1624461ba96ab4a01f9bcfc5a273fe4c99" Nov 03 15:43:46.198549 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:46.198532 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr" Nov 03 15:43:46.198703 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:46.198532 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr" event={"ID":"9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc","Type":"ContainerDied","Data":"02ead878f1d28f5df69fba0f0b7c75aa88afe790a20eeb64383ad708b9c94d2d"} Nov 03 15:43:46.207074 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:46.207057 4557 scope.go:122] "RemoveContainer" containerID="8993a109a82f6046f96ca18b47c7bfcf8d306e4cdd0db66327897f3300046f17" Nov 03 15:43:46.286121 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:46.286088 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72be94c1-dd49-4f71-8a37-6056e0939ee9" path="/var/lib/kubelet/pods/72be94c1-dd49-4f71-8a37-6056e0939ee9/volumes" Nov 03 15:43:46.286552 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:46.286537 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d60ef8-6b7d-45af-8b6b-5579fb6ce960" path="/var/lib/kubelet/pods/93d60ef8-6b7d-45af-8b6b-5579fb6ce960/volumes" Nov 03 15:43:46.365859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:46.365827 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr"] Nov 03 15:43:46.373046 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:46.373022 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-771ea0-7ff445f8bc-znqqr"] Nov 03 15:43:46.451171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:46.451040 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:46.501106 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:46.501076 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc"] Nov 03 15:43:46.505227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:46.505203 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-771ea0-6c78564585-dbcdc"] Nov 03 15:43:46.529955 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:43:46.529926 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:43:48.285001 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:48.284966 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6" path="/var/lib/kubelet/pods/8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6/volumes" Nov 03 15:43:48.285383 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:48.285350 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc" path="/var/lib/kubelet/pods/9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc/volumes" Nov 03 15:43:51.298320 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:51.298272 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:51.533483 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:51.533450 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:43:56.531311 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:43:56.529363 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:01.518309 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:01.518270 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:05.775895 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:05.775858 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:07.368444 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:07.368405 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:07.607670 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:07.607635 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:07.799565 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:07.799528 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:07.991546 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:07.991516 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:08.382316 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:08.382287 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:08.597174 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:08.597141 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:13.000378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:13.000332 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:15.264826 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:15.264786 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:15.471728 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:15.471696 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:15.682472 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:15.682433 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:16.530800 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:44:16.530765 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:44:16.785252 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:16.785163 4557 scope.go:122] "RemoveContainer" containerID="a2e63f4733f020c381d5a6003198ea5cca0a19b28024104aba9ac4f4ea74bb2f" Nov 03 15:44:18.549115 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:18.549048 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:20.054674 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:20.054640 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv"] Nov 03 15:44:20.104547 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:20.104514 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e2f4a7b-shkwv"] Nov 03 15:44:20.286634 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:20.286599 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a91a7b7b-27b0-485d-9da4-8b19173918ca" path="/var/lib/kubelet/pods/a91a7b7b-27b0-485d-9da4-8b19173918ca/volumes" Nov 03 15:44:22.086420 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:22.086380 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:22.274050 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:22.274014 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:22.715445 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:22.715414 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs"] Nov 03 15:44:22.715642 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:22.715618 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" podUID="aa0aa695-fb42-42d7-b979-55ffed84e5f1" containerName="maas-api" containerID="cri-o://06127e62cc58605130fb9e951cdda1e80a8a1fa874f90e61851487850aba09c8" gracePeriod=30 Nov 03 15:44:22.861516 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:22.861478 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:23.036922 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.036896 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:44:23.065033 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.064998 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa0aa695-fb42-42d7-b979-55ffed84e5f1-kube-api-access-72ggw\" (UniqueName: \"kubernetes.io/projected/aa0aa695-fb42-42d7-b979-55ffed84e5f1-kube-api-access-72ggw\") pod \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\" (UID: \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\") " Nov 03 15:44:23.065192 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.065049 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa0aa695-fb42-42d7-b979-55ffed84e5f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0aa695-fb42-42d7-b979-55ffed84e5f1-maas-api-tls\") pod \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\" (UID: \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\") " Nov 03 15:44:23.065192 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.065086 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa0aa695-fb42-42d7-b979-55ffed84e5f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0aa695-fb42-42d7-b979-55ffed84e5f1-openshift-service-ca\") pod \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\" (UID: \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\") " Nov 03 15:44:23.065319 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.065194 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa0aa695-fb42-42d7-b979-55ffed84e5f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0aa695-fb42-42d7-b979-55ffed84e5f1-metrics-certs\") pod \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\" (UID: \"aa0aa695-fb42-42d7-b979-55ffed84e5f1\") " Nov 03 15:44:23.065659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.065626 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa0aa695-fb42-42d7-b979-55ffed84e5f1-openshift-service-ca" pod "aa0aa695-fb42-42d7-b979-55ffed84e5f1" (UID: "aa0aa695-fb42-42d7-b979-55ffed84e5f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:44:23.065953 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.065779 4557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0aa695-fb42-42d7-b979-55ffed84e5f1-openshift-service-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:23.067344 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.067322 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa0aa695-fb42-42d7-b979-55ffed84e5f1-maas-api-tls" pod "aa0aa695-fb42-42d7-b979-55ffed84e5f1" (UID: "aa0aa695-fb42-42d7-b979-55ffed84e5f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:44:23.067695 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.067667 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa0aa695-fb42-42d7-b979-55ffed84e5f1-kube-api-access-72ggw" pod "aa0aa695-fb42-42d7-b979-55ffed84e5f1" (UID: "aa0aa695-fb42-42d7-b979-55ffed84e5f1"). InnerVolumeSpecName "kube-api-access-72ggw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:44:23.067833 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.067720 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa0aa695-fb42-42d7-b979-55ffed84e5f1-metrics-certs" pod "aa0aa695-fb42-42d7-b979-55ffed84e5f1" (UID: "aa0aa695-fb42-42d7-b979-55ffed84e5f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:44:23.169297 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.166541 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72ggw\" (UniqueName: \"kubernetes.io/projected/aa0aa695-fb42-42d7-b979-55ffed84e5f1-kube-api-access-72ggw\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:23.169297 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.166569 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0aa695-fb42-42d7-b979-55ffed84e5f1-maas-api-tls\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:23.169297 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.166595 4557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0aa695-fb42-42d7-b979-55ffed84e5f1-metrics-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:23.443049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.443013 4557 generic.go:356] "Generic (PLEG): container finished" podID="aa0aa695-fb42-42d7-b979-55ffed84e5f1" containerID="06127e62cc58605130fb9e951cdda1e80a8a1fa874f90e61851487850aba09c8" exitCode=0 Nov 03 15:44:23.443210 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.443056 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" event={"ID":"aa0aa695-fb42-42d7-b979-55ffed84e5f1","Type":"ContainerDied","Data":"06127e62cc58605130fb9e951cdda1e80a8a1fa874f90e61851487850aba09c8"} Nov 03 15:44:23.443210 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.443091 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" Nov 03 15:44:23.443210 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.443137 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs" event={"ID":"aa0aa695-fb42-42d7-b979-55ffed84e5f1","Type":"ContainerDied","Data":"218d1ca06f7e3fbc9a144bbf25b42cc775f97c42645cc0c074cc3006db89d6cf"} Nov 03 15:44:23.443210 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.443160 4557 scope.go:122] "RemoveContainer" containerID="06127e62cc58605130fb9e951cdda1e80a8a1fa874f90e61851487850aba09c8" Nov 03 15:44:23.452826 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.452808 4557 scope.go:122] "RemoveContainer" containerID="06127e62cc58605130fb9e951cdda1e80a8a1fa874f90e61851487850aba09c8" Nov 03 15:44:23.453075 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:44:23.453057 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06127e62cc58605130fb9e951cdda1e80a8a1fa874f90e61851487850aba09c8\": container with ID starting with 06127e62cc58605130fb9e951cdda1e80a8a1fa874f90e61851487850aba09c8 not found: ID does not exist" containerID="06127e62cc58605130fb9e951cdda1e80a8a1fa874f90e61851487850aba09c8" Nov 03 15:44:23.453121 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.453085 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06127e62cc58605130fb9e951cdda1e80a8a1fa874f90e61851487850aba09c8"} err="failed to get container status \"06127e62cc58605130fb9e951cdda1e80a8a1fa874f90e61851487850aba09c8\": rpc error: code = NotFound desc = could not find container \"06127e62cc58605130fb9e951cdda1e80a8a1fa874f90e61851487850aba09c8\": container with ID starting with 06127e62cc58605130fb9e951cdda1e80a8a1fa874f90e61851487850aba09c8 not found: ID does not exist" Nov 03 15:44:23.612248 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.612209 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs"] Nov 03 15:44:23.616823 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:23.616793 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-771ea0-66b75fb66d-b4dvs"] Nov 03 15:44:24.285755 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:24.285711 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa0aa695-fb42-42d7-b979-55ffed84e5f1" path="/var/lib/kubelet/pods/aa0aa695-fb42-42d7-b979-55ffed84e5f1/volumes" Nov 03 15:44:24.409169 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:24.409135 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:24.735027 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:24.734706 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:24.948206 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:24.948173 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:25.370713 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:25.370684 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:29.971955 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:29.971925 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:31.973340 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:31.973306 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:32.158263 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:32.158231 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:32.378602 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:32.378521 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:35.311883 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:35.311843 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:37.246395 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:37.246352 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:37.440285 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:37.440245 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:40.634768 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:40.634730 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:45.085202 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:45.085151 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:46.186610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:46.176059 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:46.532644 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:44:46.532559 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:44:50.382995 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.382958 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j"] Nov 03 15:44:50.383477 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.383196 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" podUID="4444e613-53d1-4286-be6e-fa3883b053ed" containerName="maas-api" containerID="cri-o://8e6334622d27f8c9893e10212c91bb2ee36a2a69a3ed6ec61247128a616ccbaa" gracePeriod=30 Nov 03 15:44:50.564347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.564134 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:50.672770 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.670540 4557 generic.go:356] "Generic (PLEG): container finished" podID="4444e613-53d1-4286-be6e-fa3883b053ed" containerID="8e6334622d27f8c9893e10212c91bb2ee36a2a69a3ed6ec61247128a616ccbaa" exitCode=0 Nov 03 15:44:50.672770 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.670598 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" event={"ID":"4444e613-53d1-4286-be6e-fa3883b053ed","Type":"ContainerDied","Data":"8e6334622d27f8c9893e10212c91bb2ee36a2a69a3ed6ec61247128a616ccbaa"} Nov 03 15:44:50.733944 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.733915 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:44:50.898929 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.898893 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4444e613-53d1-4286-be6e-fa3883b053ed-kube-api-access-zptzp\" (UniqueName: \"kubernetes.io/projected/4444e613-53d1-4286-be6e-fa3883b053ed-kube-api-access-zptzp\") pod \"4444e613-53d1-4286-be6e-fa3883b053ed\" (UID: \"4444e613-53d1-4286-be6e-fa3883b053ed\") " Nov 03 15:44:50.899112 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.898952 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4444e613-53d1-4286-be6e-fa3883b053ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4444e613-53d1-4286-be6e-fa3883b053ed-metrics-certs\") pod \"4444e613-53d1-4286-be6e-fa3883b053ed\" (UID: \"4444e613-53d1-4286-be6e-fa3883b053ed\") " Nov 03 15:44:50.899112 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.899082 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4444e613-53d1-4286-be6e-fa3883b053ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4444e613-53d1-4286-be6e-fa3883b053ed-openshift-service-ca\") pod \"4444e613-53d1-4286-be6e-fa3883b053ed\" (UID: \"4444e613-53d1-4286-be6e-fa3883b053ed\") " Nov 03 15:44:50.899244 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.899229 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4444e613-53d1-4286-be6e-fa3883b053ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4444e613-53d1-4286-be6e-fa3883b053ed-maas-api-tls\") pod \"4444e613-53d1-4286-be6e-fa3883b053ed\" (UID: \"4444e613-53d1-4286-be6e-fa3883b053ed\") " Nov 03 15:44:50.899482 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.899446 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4444e613-53d1-4286-be6e-fa3883b053ed-openshift-service-ca" pod "4444e613-53d1-4286-be6e-fa3883b053ed" (UID: "4444e613-53d1-4286-be6e-fa3883b053ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:44:50.899757 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.899736 4557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4444e613-53d1-4286-be6e-fa3883b053ed-openshift-service-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:50.901128 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.901102 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4444e613-53d1-4286-be6e-fa3883b053ed-metrics-certs" pod "4444e613-53d1-4286-be6e-fa3883b053ed" (UID: "4444e613-53d1-4286-be6e-fa3883b053ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:44:50.901214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.901162 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4444e613-53d1-4286-be6e-fa3883b053ed-kube-api-access-zptzp" pod "4444e613-53d1-4286-be6e-fa3883b053ed" (UID: "4444e613-53d1-4286-be6e-fa3883b053ed"). InnerVolumeSpecName "kube-api-access-zptzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:44:50.901214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.901175 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4444e613-53d1-4286-be6e-fa3883b053ed-maas-api-tls" pod "4444e613-53d1-4286-be6e-fa3883b053ed" (UID: "4444e613-53d1-4286-be6e-fa3883b053ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:44:50.934740 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.934678 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9"] Nov 03 15:44:50.935231 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935209 4557 cpu_manager.go:414] "containerMap: removing container" podUID="93d60ef8-6b7d-45af-8b6b-5579fb6ce960" containerName="istio-proxy" Nov 03 15:44:50.935231 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935227 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d60ef8-6b7d-45af-8b6b-5579fb6ce960" containerName="istio-proxy" Nov 03 15:44:50.935442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935251 4557 cpu_manager.go:414] "containerMap: removing container" podUID="8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6" containerName="payload-processing" Nov 03 15:44:50.935442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935259 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6" containerName="payload-processing" Nov 03 15:44:50.935442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935276 4557 cpu_manager.go:414] "containerMap: removing container" podUID="9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc" containerName="payload-pre-processing" Nov 03 15:44:50.935442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935284 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc" containerName="payload-pre-processing" Nov 03 15:44:50.935442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935305 4557 cpu_manager.go:414] "containerMap: removing container" podUID="4444e613-53d1-4286-be6e-fa3883b053ed" containerName="maas-api" Nov 03 15:44:50.935442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935329 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4444e613-53d1-4286-be6e-fa3883b053ed" containerName="maas-api" Nov 03 15:44:50.935442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935343 4557 cpu_manager.go:414] "containerMap: removing container" podUID="72be94c1-dd49-4f71-8a37-6056e0939ee9" containerName="istio-proxy" Nov 03 15:44:50.935442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935349 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72be94c1-dd49-4f71-8a37-6056e0939ee9" containerName="istio-proxy" Nov 03 15:44:50.935442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935363 4557 cpu_manager.go:414] "containerMap: removing container" podUID="67217abc-0f32-4031-ac20-5ef4293cc64e" containerName="payload-processing" Nov 03 15:44:50.935442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935371 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67217abc-0f32-4031-ac20-5ef4293cc64e" containerName="payload-processing" Nov 03 15:44:50.935442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935383 4557 cpu_manager.go:414] "containerMap: removing container" podUID="795e739f-2842-4d1b-bbd7-5770099cc7bb" containerName="payload-pre-processing" Nov 03 15:44:50.935442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935390 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795e739f-2842-4d1b-bbd7-5770099cc7bb" containerName="payload-pre-processing" Nov 03 15:44:50.935442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935409 4557 cpu_manager.go:414] "containerMap: removing container" podUID="aa0aa695-fb42-42d7-b979-55ffed84e5f1" containerName="maas-api" Nov 03 15:44:50.935442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935416 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa0aa695-fb42-42d7-b979-55ffed84e5f1" containerName="maas-api" Nov 03 15:44:50.936168 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935537 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa0aa695-fb42-42d7-b979-55ffed84e5f1" containerName="maas-api" Nov 03 15:44:50.936168 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935556 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="9716ef2e-74f1-44a1-a9dd-a5252b7d9dfc" containerName="payload-pre-processing" Nov 03 15:44:50.936168 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935569 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="4444e613-53d1-4286-be6e-fa3883b053ed" containerName="maas-api" Nov 03 15:44:50.936168 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935592 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="72be94c1-dd49-4f71-8a37-6056e0939ee9" containerName="istio-proxy" Nov 03 15:44:50.936168 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935600 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d60ef8-6b7d-45af-8b6b-5579fb6ce960" containerName="istio-proxy" Nov 03 15:44:50.936168 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935609 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="67217abc-0f32-4031-ac20-5ef4293cc64e" containerName="payload-processing" Nov 03 15:44:50.936168 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935617 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="795e739f-2842-4d1b-bbd7-5770099cc7bb" containerName="payload-pre-processing" Nov 03 15:44:50.936168 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.935629 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="8666a3f9-e3aa-41ae-b1c2-ffdeed9d80c6" containerName="payload-processing" Nov 03 15:44:50.939124 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.939106 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:50.949321 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.949299 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc"] Nov 03 15:44:50.953083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:50.953043 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.001121 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.001087 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4444e613-53d1-4286-be6e-fa3883b053ed-maas-api-tls\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:51.001121 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.001122 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zptzp\" (UniqueName: \"kubernetes.io/projected/4444e613-53d1-4286-be6e-fa3883b053ed-kube-api-access-zptzp\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:51.001341 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.001139 4557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4444e613-53d1-4286-be6e-fa3883b053ed-metrics-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:44:51.025950 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.025917 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9"] Nov 03 15:44:51.049948 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.049921 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc"] Nov 03 15:44:51.101922 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.101890 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.102091 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.101926 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.102091 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.101953 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhtds\" (UniqueName: \"kubernetes.io/projected/4386210a-32ea-4fe8-b342-63c4cc68855b-kube-api-access-hhtds\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.102188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.102089 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.102188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.102118 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.102188 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.102144 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.102305 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.102246 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sslfc\" (UniqueName: \"kubernetes.io/projected/f23b9ae0-fb87-47e3-b226-198a656ae1e4-kube-api-access-sslfc\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.102305 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.102275 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.102383 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.102303 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.102383 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.102322 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.102447 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.102384 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4386210a-32ea-4fe8-b342-63c4cc68855b-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.102447 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.102404 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.102447 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.102422 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.102562 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.102476 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.102562 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.102495 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.102562 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.102531 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.102732 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.102622 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.102732 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.102668 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.204067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.203986 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.204067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204024 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.204067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204059 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4386210a-32ea-4fe8-b342-63c4cc68855b-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.204347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204089 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.204347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204114 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.204347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204183 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.204347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204205 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.204347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204235 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.204347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204258 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.204347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204293 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.204753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204369 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.204753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204402 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.204753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204444 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.204753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204449 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhtds\" (UniqueName: \"kubernetes.io/projected/4386210a-32ea-4fe8-b342-63c4cc68855b-kube-api-access-hhtds\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.204753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204499 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.204753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204561 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.204753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204608 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.204753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204648 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.204753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204699 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sslfc\" (UniqueName: \"kubernetes.io/projected/f23b9ae0-fb87-47e3-b226-198a656ae1e4-kube-api-access-sslfc\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.204753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204727 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.204753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204751 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.205432 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204783 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.205432 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204925 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4386210a-32ea-4fe8-b342-63c4cc68855b-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.205432 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.204936 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.205432 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.205214 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.205432 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.205266 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.205432 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.205412 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.206061 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.205615 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.207346 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.207322 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.207529 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.207511 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.207790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.207773 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.207887 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.207793 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.268163 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.268135 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sslfc\" (UniqueName: \"kubernetes.io/projected/f23b9ae0-fb87-47e3-b226-198a656ae1e4-kube-api-access-sslfc\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.268296 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.268182 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.269042 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.269020 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhtds\" (UniqueName: \"kubernetes.io/projected/4386210a-32ea-4fe8-b342-63c4cc68855b-kube-api-access-hhtds\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.269359 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.269342 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.553322 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.553289 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:51.565722 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.565691 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:51.677445 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.677408 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" event={"ID":"4444e613-53d1-4286-be6e-fa3883b053ed","Type":"ContainerDied","Data":"5d862348a562296ef94e7862dd7e1ddd1f053105e2be4ab9533516ca35c39907"} Nov 03 15:44:51.677638 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.677484 4557 scope.go:122] "RemoveContainer" containerID="8e6334622d27f8c9893e10212c91bb2ee36a2a69a3ed6ec61247128a616ccbaa" Nov 03 15:44:51.677711 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.677650 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j" Nov 03 15:44:51.724772 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.724689 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:51.866629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.866271 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j"] Nov 03 15:44:51.867276 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.867248 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5d3cde-78b7cbd56f-lxp2j"] Nov 03 15:44:51.952823 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.952794 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:51.982212 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:44:51.982181 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23b9ae0_fb87_47e3_b226_198a656ae1e4.slice/crio-2838c3c83f090417009b03e6eefae3ce556eb91adce8970cca6d7ef7f6696bb7 WatchSource:0}: Error finding container 2838c3c83f090417009b03e6eefae3ce556eb91adce8970cca6d7ef7f6696bb7: Status 404 returned error can't find the container with id 2838c3c83f090417009b03e6eefae3ce556eb91adce8970cca6d7ef7f6696bb7 Nov 03 15:44:51.987822 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.985036 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:44:51.987822 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.985213 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:44:51.987822 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.985294 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:44:51.996758 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:51.996729 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc"] Nov 03 15:44:52.024825 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:52.024792 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9"] Nov 03 15:44:52.035817 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:44:52.035787 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4386210a_32ea_4fe8_b342_63c4cc68855b.slice/crio-96280fb1467ad6e19729ccf446e5225a78da5e5c566970701d7da5e189586f2b WatchSource:0}: Error finding container 96280fb1467ad6e19729ccf446e5225a78da5e5c566970701d7da5e189586f2b: Status 404 returned error can't find the container with id 96280fb1467ad6e19729ccf446e5225a78da5e5c566970701d7da5e189586f2b Nov 03 15:44:52.038211 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:52.038182 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:44:52.038316 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:52.038255 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:44:52.038316 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:52.038297 4557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:44:52.176956 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:52.176871 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:52.289102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:52.289053 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4444e613-53d1-4286-be6e-fa3883b053ed" path="/var/lib/kubelet/pods/4444e613-53d1-4286-be6e-fa3883b053ed/volumes" Nov 03 15:44:52.400607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:52.397716 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:52.597313 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:52.597226 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:52.728807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:52.728045 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" event={"ID":"4386210a-32ea-4fe8-b342-63c4cc68855b","Type":"ContainerStarted","Data":"e249d06edbe300792cd0555e18253247a006c4aca0a93cafcc3c00b8420343a2"} Nov 03 15:44:52.728807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:52.728111 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" event={"ID":"4386210a-32ea-4fe8-b342-63c4cc68855b","Type":"ContainerStarted","Data":"96280fb1467ad6e19729ccf446e5225a78da5e5c566970701d7da5e189586f2b"} Nov 03 15:44:52.730748 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:52.730722 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" event={"ID":"f23b9ae0-fb87-47e3-b226-198a656ae1e4","Type":"ContainerStarted","Data":"e048164617d60d4efe40bc662512ce2d0ae390b76ca5a80f0a38e7f9452f514c"} Nov 03 15:44:52.730856 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:52.730774 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" event={"ID":"f23b9ae0-fb87-47e3-b226-198a656ae1e4","Type":"ContainerStarted","Data":"2838c3c83f090417009b03e6eefae3ce556eb91adce8970cca6d7ef7f6696bb7"} Nov 03 15:44:52.859658 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:52.858967 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" podStartSLOduration=2.858943884 podStartE2EDuration="2.858943884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:44:52.857037458 +0000 UTC m=+2202.700159001" watchObservedRunningTime="2026-11-03 15:44:52.858943884 +0000 UTC m=+2202.702065427" Nov 03 15:44:53.094605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:53.076551 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:53.553974 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:53.553938 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:53.555369 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:53.555345 4557 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 15:44:53.555490 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:53.555398 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" podUID="4386210a-32ea-4fe8-b342-63c4cc68855b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 15:44:53.566646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:53.566623 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:53.567723 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:53.567706 4557 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 03 15:44:53.567859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:53.567841 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" podUID="f23b9ae0-fb87-47e3-b226-198a656ae1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 03 15:44:54.553926 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:54.553886 4557 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 15:44:54.554332 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:54.553955 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" podUID="4386210a-32ea-4fe8-b342-63c4cc68855b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 15:44:54.566755 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:54.566724 4557 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 03 15:44:54.566884 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:54.566772 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" podUID="f23b9ae0-fb87-47e3-b226-198a656ae1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 03 15:44:55.555133 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:55.555052 4557 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 15:44:55.555133 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:55.555109 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" podUID="4386210a-32ea-4fe8-b342-63c4cc68855b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 15:44:55.566251 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:55.566228 4557 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 03 15:44:55.566409 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:55.566392 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" podUID="f23b9ae0-fb87-47e3-b226-198a656ae1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 03 15:44:56.553726 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:56.553690 4557 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 15:44:56.553942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:56.553750 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" podUID="4386210a-32ea-4fe8-b342-63c4cc68855b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 15:44:56.566399 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:56.566370 4557 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 03 15:44:56.566749 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:56.566409 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" podUID="f23b9ae0-fb87-47e3-b226-198a656ae1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 03 15:44:57.496824 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:57.496787 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:44:57.554992 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:57.554957 4557 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 15:44:57.555278 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:57.555246 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" podUID="4386210a-32ea-4fe8-b342-63c4cc68855b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 15:44:57.568128 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:57.567864 4557 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 03 15:44:57.568128 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:57.567914 4557 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" podUID="f23b9ae0-fb87-47e3-b226-198a656ae1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 03 15:44:58.609364 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:58.609335 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:58.610313 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:58.610290 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:58.611679 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:58.611658 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:44:58.641437 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:58.641410 4557 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:58.642024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:58.642001 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:58.642904 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:58.642885 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:44:58.781099 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:44:58.781053 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" podStartSLOduration=8.781040834 podStartE2EDuration="8.781040834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:44:52.998885389 +0000 UTC m=+2202.842006965" watchObservedRunningTime="2026-11-03 15:44:58.781040834 +0000 UTC m=+2208.624162374" Nov 03 15:45:00.177829 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.177797 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7"] Nov 03 15:45:00.182766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.182743 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw"] Nov 03 15:45:00.182895 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.182864 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" Nov 03 15:45:00.186725 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.186705 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k"] Nov 03 15:45:00.186865 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.186821 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" Nov 03 15:45:00.190156 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.190135 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" Nov 03 15:45:00.234964 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.234929 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw"] Nov 03 15:45:00.241884 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.241856 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs"] Nov 03 15:45:00.242025 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.241955 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" Nov 03 15:45:00.246867 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.246843 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc"] Nov 03 15:45:00.246990 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.246932 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" Nov 03 15:45:00.250766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.250738 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" Nov 03 15:45:00.264841 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.264811 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:00.285723 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.285690 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:45:00.294629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.294093 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lkcn\" (UniqueName: \"kubernetes.io/projected/a177f815-6766-447d-afea-c04c1a5dc18d-kube-api-access-4lkcn\") pod \"maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw\" (UID: \"a177f815-6766-447d-afea-c04c1a5dc18d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" Nov 03 15:45:00.294629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.294223 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9xxw\" (UniqueName: \"kubernetes.io/projected/148ee23d-c6d4-42b6-9173-e16173fe1063-kube-api-access-x9xxw\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw\" (UID: \"148ee23d-c6d4-42b6-9173-e16173fe1063\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" Nov 03 15:45:00.294629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.294273 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/148ee23d-c6d4-42b6-9173-e16173fe1063-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw\" (UID: \"148ee23d-c6d4-42b6-9173-e16173fe1063\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" Nov 03 15:45:00.294629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.294311 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvpkk\" (UniqueName: \"kubernetes.io/projected/43aaa640-044f-43cd-ab66-089ed62e48fb-kube-api-access-kvpkk\") pod \"maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k\" (UID: \"43aaa640-044f-43cd-ab66-089ed62e48fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" Nov 03 15:45:00.294629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.294339 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzp2s\" (UniqueName: \"kubernetes.io/projected/6120a8ff-bed0-4512-bd06-09e7e917a26c-kube-api-access-fzp2s\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7\" (UID: \"6120a8ff-bed0-4512-bd06-09e7e917a26c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" Nov 03 15:45:00.294629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.294402 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a177f815-6766-447d-afea-c04c1a5dc18d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw\" (UID: \"a177f815-6766-447d-afea-c04c1a5dc18d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" Nov 03 15:45:00.294629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.294437 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aedd62-09e8-4bf9-aa75-fab98c1cf785-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895345-4zwrs\" (UID: \"82aedd62-09e8-4bf9-aa75-fab98c1cf785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" Nov 03 15:45:00.294629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.294462 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f53996-4386-4d7f-be27-4730e4804945-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc\" (UID: \"c6f53996-4386-4d7f-be27-4730e4804945\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" Nov 03 15:45:00.294629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.294495 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43aaa640-044f-43cd-ab66-089ed62e48fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k\" (UID: \"43aaa640-044f-43cd-ab66-089ed62e48fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" Nov 03 15:45:00.294629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.294521 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqzj4\" (UniqueName: \"kubernetes.io/projected/c6f53996-4386-4d7f-be27-4730e4804945-kube-api-access-hqzj4\") pod \"maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc\" (UID: \"c6f53996-4386-4d7f-be27-4730e4804945\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" Nov 03 15:45:00.294629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.294598 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x86kb\" (UniqueName: \"kubernetes.io/projected/82aedd62-09e8-4bf9-aa75-fab98c1cf785-kube-api-access-x86kb\") pod \"maas-api-key-cleanup-29895345-4zwrs\" (UID: \"82aedd62-09e8-4bf9-aa75-fab98c1cf785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" Nov 03 15:45:00.307984 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.294670 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6120a8ff-bed0-4512-bd06-09e7e917a26c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7\" (UID: \"6120a8ff-bed0-4512-bd06-09e7e917a26c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" Nov 03 15:45:00.328496 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.327948 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7"] Nov 03 15:45:00.328496 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.328009 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k"] Nov 03 15:45:00.328496 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.328038 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw"] Nov 03 15:45:00.328496 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.328062 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs"] Nov 03 15:45:00.330052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.330027 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc"] Nov 03 15:45:00.333246 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.333221 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw"] Nov 03 15:45:00.396181 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.396151 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a177f815-6766-447d-afea-c04c1a5dc18d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw\" (UID: \"a177f815-6766-447d-afea-c04c1a5dc18d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" Nov 03 15:45:00.396181 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.396187 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aedd62-09e8-4bf9-aa75-fab98c1cf785-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895345-4zwrs\" (UID: \"82aedd62-09e8-4bf9-aa75-fab98c1cf785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" Nov 03 15:45:00.396420 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.396204 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f53996-4386-4d7f-be27-4730e4804945-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc\" (UID: \"c6f53996-4386-4d7f-be27-4730e4804945\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" Nov 03 15:45:00.396420 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.396233 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43aaa640-044f-43cd-ab66-089ed62e48fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k\" (UID: \"43aaa640-044f-43cd-ab66-089ed62e48fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" Nov 03 15:45:00.396420 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.396264 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqzj4\" (UniqueName: \"kubernetes.io/projected/c6f53996-4386-4d7f-be27-4730e4804945-kube-api-access-hqzj4\") pod \"maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc\" (UID: \"c6f53996-4386-4d7f-be27-4730e4804945\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" Nov 03 15:45:00.396420 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.396322 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x86kb\" (UniqueName: \"kubernetes.io/projected/82aedd62-09e8-4bf9-aa75-fab98c1cf785-kube-api-access-x86kb\") pod \"maas-api-key-cleanup-29895345-4zwrs\" (UID: \"82aedd62-09e8-4bf9-aa75-fab98c1cf785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" Nov 03 15:45:00.396420 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.396401 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6120a8ff-bed0-4512-bd06-09e7e917a26c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7\" (UID: \"6120a8ff-bed0-4512-bd06-09e7e917a26c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" Nov 03 15:45:00.396725 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.396442 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lkcn\" (UniqueName: \"kubernetes.io/projected/a177f815-6766-447d-afea-c04c1a5dc18d-kube-api-access-4lkcn\") pod \"maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw\" (UID: \"a177f815-6766-447d-afea-c04c1a5dc18d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" Nov 03 15:45:00.396725 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.396594 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9xxw\" (UniqueName: \"kubernetes.io/projected/148ee23d-c6d4-42b6-9173-e16173fe1063-kube-api-access-x9xxw\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw\" (UID: \"148ee23d-c6d4-42b6-9173-e16173fe1063\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" Nov 03 15:45:00.396725 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.396648 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/148ee23d-c6d4-42b6-9173-e16173fe1063-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw\" (UID: \"148ee23d-c6d4-42b6-9173-e16173fe1063\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" Nov 03 15:45:00.396725 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.396679 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvpkk\" (UniqueName: \"kubernetes.io/projected/43aaa640-044f-43cd-ab66-089ed62e48fb-kube-api-access-kvpkk\") pod \"maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k\" (UID: \"43aaa640-044f-43cd-ab66-089ed62e48fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" Nov 03 15:45:00.396725 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.396711 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzp2s\" (UniqueName: \"kubernetes.io/projected/6120a8ff-bed0-4512-bd06-09e7e917a26c-kube-api-access-fzp2s\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7\" (UID: \"6120a8ff-bed0-4512-bd06-09e7e917a26c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" Nov 03 15:45:00.397130 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.397107 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f53996-4386-4d7f-be27-4730e4804945-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc\" (UID: \"c6f53996-4386-4d7f-be27-4730e4804945\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" Nov 03 15:45:00.397200 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.397133 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43aaa640-044f-43cd-ab66-089ed62e48fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k\" (UID: \"43aaa640-044f-43cd-ab66-089ed62e48fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" Nov 03 15:45:00.397200 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.397133 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a177f815-6766-447d-afea-c04c1a5dc18d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw\" (UID: \"a177f815-6766-447d-afea-c04c1a5dc18d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" Nov 03 15:45:00.397200 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.397168 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aedd62-09e8-4bf9-aa75-fab98c1cf785-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895345-4zwrs\" (UID: \"82aedd62-09e8-4bf9-aa75-fab98c1cf785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" Nov 03 15:45:00.397408 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.397390 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6120a8ff-bed0-4512-bd06-09e7e917a26c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7\" (UID: \"6120a8ff-bed0-4512-bd06-09e7e917a26c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" Nov 03 15:45:00.397552 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.397534 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/148ee23d-c6d4-42b6-9173-e16173fe1063-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw\" (UID: \"148ee23d-c6d4-42b6-9173-e16173fe1063\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" Nov 03 15:45:00.459126 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.459059 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqzj4\" (UniqueName: \"kubernetes.io/projected/c6f53996-4386-4d7f-be27-4730e4804945-kube-api-access-hqzj4\") pod \"maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc\" (UID: \"c6f53996-4386-4d7f-be27-4730e4804945\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" Nov 03 15:45:00.459949 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.459922 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x86kb\" (UniqueName: \"kubernetes.io/projected/82aedd62-09e8-4bf9-aa75-fab98c1cf785-kube-api-access-x86kb\") pod \"maas-api-key-cleanup-29895345-4zwrs\" (UID: \"82aedd62-09e8-4bf9-aa75-fab98c1cf785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" Nov 03 15:45:00.460105 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.460086 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzp2s\" (UniqueName: \"kubernetes.io/projected/6120a8ff-bed0-4512-bd06-09e7e917a26c-kube-api-access-fzp2s\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7\" (UID: \"6120a8ff-bed0-4512-bd06-09e7e917a26c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" Nov 03 15:45:00.460567 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.460543 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvpkk\" (UniqueName: \"kubernetes.io/projected/43aaa640-044f-43cd-ab66-089ed62e48fb-kube-api-access-kvpkk\") pod \"maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k\" (UID: \"43aaa640-044f-43cd-ab66-089ed62e48fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" Nov 03 15:45:00.460681 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.460627 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lkcn\" (UniqueName: \"kubernetes.io/projected/a177f815-6766-447d-afea-c04c1a5dc18d-kube-api-access-4lkcn\") pod \"maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw\" (UID: \"a177f815-6766-447d-afea-c04c1a5dc18d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" Nov 03 15:45:00.463296 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.463278 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9xxw\" (UniqueName: \"kubernetes.io/projected/148ee23d-c6d4-42b6-9173-e16173fe1063-kube-api-access-x9xxw\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw\" (UID: \"148ee23d-c6d4-42b6-9173-e16173fe1063\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" Nov 03 15:45:00.468212 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.468188 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:00.499771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.499751 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" Nov 03 15:45:00.507063 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.507049 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" Nov 03 15:45:00.512961 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.512939 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" Nov 03 15:45:00.562315 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.561793 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" Nov 03 15:45:00.574992 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.573846 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" Nov 03 15:45:00.609046 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.608814 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" Nov 03 15:45:00.660878 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:00.660810 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:01.109960 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.109919 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7"] Nov 03 15:45:01.110090 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:45:01.110062 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6120a8ff_bed0_4512_bd06_09e7e917a26c.slice/crio-e2205b87d06b49bf8bf3ea01529a85a282db5b1e1e0fed60cb2386a364bbef28 WatchSource:0}: Error finding container e2205b87d06b49bf8bf3ea01529a85a282db5b1e1e0fed60cb2386a364bbef28: Status 404 returned error can't find the container with id e2205b87d06b49bf8bf3ea01529a85a282db5b1e1e0fed60cb2386a364bbef28 Nov 03 15:45:01.156334 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:45:01.156310 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43aaa640_044f_43cd_ab66_089ed62e48fb.slice/crio-2280dc36a8cd41a89b1b4d8089979c1c75fdbb323fa18a2f62c3210d1579d97a WatchSource:0}: Error finding container 2280dc36a8cd41a89b1b4d8089979c1c75fdbb323fa18a2f62c3210d1579d97a: Status 404 returned error can't find the container with id 2280dc36a8cd41a89b1b4d8089979c1c75fdbb323fa18a2f62c3210d1579d97a Nov 03 15:45:01.156641 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.156615 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw"] Nov 03 15:45:01.157812 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:45:01.157790 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda177f815_6766_447d_afea_c04c1a5dc18d.slice/crio-7241a1727f9340ce243ab5c0bda94ea21a5d3ff46db110f8851386bb5a20852d WatchSource:0}: Error finding container 7241a1727f9340ce243ab5c0bda94ea21a5d3ff46db110f8851386bb5a20852d: Status 404 returned error can't find the container with id 7241a1727f9340ce243ab5c0bda94ea21a5d3ff46db110f8851386bb5a20852d Nov 03 15:45:01.159782 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.159755 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k"] Nov 03 15:45:01.219260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.219233 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc"] Nov 03 15:45:01.220461 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:45:01.220434 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148ee23d_c6d4_42b6_9173_e16173fe1063.slice/crio-1d6ea5c8599930e8eed54b7606ab3d7c8005235dd66bf20252d3258c74def14b WatchSource:0}: Error finding container 1d6ea5c8599930e8eed54b7606ab3d7c8005235dd66bf20252d3258c74def14b: Status 404 returned error can't find the container with id 1d6ea5c8599930e8eed54b7606ab3d7c8005235dd66bf20252d3258c74def14b Nov 03 15:45:01.221239 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:45:01.221214 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f53996_4386_4d7f_be27_4730e4804945.slice/crio-7e8811cf2583e21e2be03669a7b202312a188c2f71263e389bd3df7031575727 WatchSource:0}: Error finding container 7e8811cf2583e21e2be03669a7b202312a188c2f71263e389bd3df7031575727: Status 404 returned error can't find the container with id 7e8811cf2583e21e2be03669a7b202312a188c2f71263e389bd3df7031575727 Nov 03 15:45:01.222403 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.222379 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw"] Nov 03 15:45:01.236859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.236835 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs"] Nov 03 15:45:01.237135 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:45:01.237109 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82aedd62_09e8_4bf9_aa75_fab98c1cf785.slice/crio-c13ebb756b283c37c267cf985ae7c95139bc91b1466950d2e6edbb5ff9700e62 WatchSource:0}: Error finding container c13ebb756b283c37c267cf985ae7c95139bc91b1466950d2e6edbb5ff9700e62: Status 404 returned error can't find the container with id c13ebb756b283c37c267cf985ae7c95139bc91b1466950d2e6edbb5ff9700e62 Nov 03 15:45:01.608152 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.608122 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq"] Nov 03 15:45:01.611888 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.611871 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq" Nov 03 15:45:01.673400 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.673321 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:45:01.674175 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.674143 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq"] Nov 03 15:45:01.737862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.737819 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce046a64-2b99-43e2-a354-a4129e45af72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-90fcca-54zhq\" (UID: \"ce046a64-2b99-43e2-a354-a4129e45af72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq" Nov 03 15:45:01.738029 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.737961 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce046a64-2b99-43e2-a354-a4129e45af72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-90fcca-54zhq\" (UID: \"ce046a64-2b99-43e2-a354-a4129e45af72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq" Nov 03 15:45:01.738092 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.738072 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwr5v\" (UniqueName: \"kubernetes.io/projected/ce046a64-2b99-43e2-a354-a4129e45af72-kube-api-access-pwr5v\") pod \"maas-api-revoke-keys-e2e-api-b-90fcca-54zhq\" (UID: \"ce046a64-2b99-43e2-a354-a4129e45af72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq" Nov 03 15:45:01.839396 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.839365 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce046a64-2b99-43e2-a354-a4129e45af72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-90fcca-54zhq\" (UID: \"ce046a64-2b99-43e2-a354-a4129e45af72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq" Nov 03 15:45:01.840394 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.839434 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwr5v\" (UniqueName: \"kubernetes.io/projected/ce046a64-2b99-43e2-a354-a4129e45af72-kube-api-access-pwr5v\") pod \"maas-api-revoke-keys-e2e-api-b-90fcca-54zhq\" (UID: \"ce046a64-2b99-43e2-a354-a4129e45af72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq" Nov 03 15:45:01.840394 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.839489 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce046a64-2b99-43e2-a354-a4129e45af72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-90fcca-54zhq\" (UID: \"ce046a64-2b99-43e2-a354-a4129e45af72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq" Nov 03 15:45:01.840394 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.840295 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce046a64-2b99-43e2-a354-a4129e45af72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-90fcca-54zhq\" (UID: \"ce046a64-2b99-43e2-a354-a4129e45af72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq" Nov 03 15:45:01.842145 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.842121 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce046a64-2b99-43e2-a354-a4129e45af72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-90fcca-54zhq\" (UID: \"ce046a64-2b99-43e2-a354-a4129e45af72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq" Nov 03 15:45:01.885424 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.885273 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" event={"ID":"a177f815-6766-447d-afea-c04c1a5dc18d","Type":"ContainerStarted","Data":"8de70d3a7480b1ded7a7ea1cbdec6f4468ed2a2776af55f64e32c3aca9a017c0"} Nov 03 15:45:01.885424 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.885359 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" event={"ID":"a177f815-6766-447d-afea-c04c1a5dc18d","Type":"ContainerStarted","Data":"7241a1727f9340ce243ab5c0bda94ea21a5d3ff46db110f8851386bb5a20852d"} Nov 03 15:45:01.887165 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.887131 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" event={"ID":"82aedd62-09e8-4bf9-aa75-fab98c1cf785","Type":"ContainerStarted","Data":"b41e2414ae3bf00715456e22116f861b488e3c20e8bee2c5402f818e9bfd1487"} Nov 03 15:45:01.887281 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.887189 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" event={"ID":"82aedd62-09e8-4bf9-aa75-fab98c1cf785","Type":"ContainerStarted","Data":"c13ebb756b283c37c267cf985ae7c95139bc91b1466950d2e6edbb5ff9700e62"} Nov 03 15:45:01.888917 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.888870 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" event={"ID":"c6f53996-4386-4d7f-be27-4730e4804945","Type":"ContainerStarted","Data":"084c299025219540e6d7e37caa3ca709b8f1a86a8b0ddac274629ff48f4606ae"} Nov 03 15:45:01.889058 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.888949 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" event={"ID":"c6f53996-4386-4d7f-be27-4730e4804945","Type":"ContainerStarted","Data":"7e8811cf2583e21e2be03669a7b202312a188c2f71263e389bd3df7031575727"} Nov 03 15:45:01.890297 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.890278 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" event={"ID":"43aaa640-044f-43cd-ab66-089ed62e48fb","Type":"ContainerStarted","Data":"f0cf8c417ce1d115e41f8a7c51fb8d7210cd87ec2e8cb5cc5fb0bdc7e30eb650"} Nov 03 15:45:01.890418 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.890316 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" event={"ID":"43aaa640-044f-43cd-ab66-089ed62e48fb","Type":"ContainerStarted","Data":"2280dc36a8cd41a89b1b4d8089979c1c75fdbb323fa18a2f62c3210d1579d97a"} Nov 03 15:45:01.891784 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.891753 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" event={"ID":"6120a8ff-bed0-4512-bd06-09e7e917a26c","Type":"ContainerStarted","Data":"cb6eefe6da4d0273989e40cd7fa01004a5e56acfdadc021f6146e9a92f98e891"} Nov 03 15:45:01.891907 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.891792 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" event={"ID":"6120a8ff-bed0-4512-bd06-09e7e917a26c","Type":"ContainerStarted","Data":"e2205b87d06b49bf8bf3ea01529a85a282db5b1e1e0fed60cb2386a364bbef28"} Nov 03 15:45:01.893317 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.893295 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" event={"ID":"148ee23d-c6d4-42b6-9173-e16173fe1063","Type":"ContainerStarted","Data":"2c142cea75e013781a13a56bc68e8e11470a5c8dfc3869ee530166f6261ea439"} Nov 03 15:45:01.893412 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.893331 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" event={"ID":"148ee23d-c6d4-42b6-9173-e16173fe1063","Type":"ContainerStarted","Data":"1d6ea5c8599930e8eed54b7606ab3d7c8005235dd66bf20252d3258c74def14b"} Nov 03 15:45:01.905115 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.905093 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwr5v\" (UniqueName: \"kubernetes.io/projected/ce046a64-2b99-43e2-a354-a4129e45af72-kube-api-access-pwr5v\") pod \"maas-api-revoke-keys-e2e-api-b-90fcca-54zhq\" (UID: \"ce046a64-2b99-43e2-a354-a4129e45af72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq" Nov 03 15:45:01.924314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:01.924262 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq" Nov 03 15:45:02.005535 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.005480 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" podStartSLOduration=2.005466167 podStartE2EDuration="2.005466167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:02.003332937 +0000 UTC m=+2211.846454479" watchObservedRunningTime="2026-11-03 15:45:02.005466167 +0000 UTC m=+2211.848587709" Nov 03 15:45:02.153410 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.153351 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" podStartSLOduration=2.1533338 podStartE2EDuration="2.1533338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:02.150134874 +0000 UTC m=+2211.993256415" watchObservedRunningTime="2026-11-03 15:45:02.1533338 +0000 UTC m=+2211.996455344" Nov 03 15:45:02.296434 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.296378 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" podStartSLOduration=2.296364935 podStartE2EDuration="2.296364935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:02.294213349 +0000 UTC m=+2212.137334889" watchObservedRunningTime="2026-11-03 15:45:02.296364935 +0000 UTC m=+2212.139486476" Nov 03 15:45:02.338713 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.338687 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq"] Nov 03 15:45:02.340077 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:45:02.340054 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce046a64_2b99_43e2_a354_a4129e45af72.slice/crio-19c19b06c697c14e80f4149d257006749215d620e7ead089963b83fb3721d57c WatchSource:0}: Error finding container 19c19b06c697c14e80f4149d257006749215d620e7ead089963b83fb3721d57c: Status 404 returned error can't find the container with id 19c19b06c697c14e80f4149d257006749215d620e7ead089963b83fb3721d57c Nov 03 15:45:02.435751 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.435650 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" podStartSLOduration=2.43563776 podStartE2EDuration="2.43563776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:02.433175075 +0000 UTC m=+2212.276296618" watchObservedRunningTime="2026-11-03 15:45:02.43563776 +0000 UTC m=+2212.278759339" Nov 03 15:45:02.552987 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.552921 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" podStartSLOduration=2.552902583 podStartE2EDuration="2.552902583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:02.552280947 +0000 UTC m=+2212.395402518" watchObservedRunningTime="2026-11-03 15:45:02.552902583 +0000 UTC m=+2212.396024128" Nov 03 15:45:02.672914 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.672857 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" podStartSLOduration=2.6728416189999997 podStartE2EDuration="2.672841619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:02.668275666 +0000 UTC m=+2212.511397206" watchObservedRunningTime="2026-11-03 15:45:02.672841619 +0000 UTC m=+2212.515963164" Nov 03 15:45:02.789638 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.789487 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:02.900169 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.900134 4557 generic.go:356] "Generic (PLEG): container finished" podID="148ee23d-c6d4-42b6-9173-e16173fe1063" containerID="2c142cea75e013781a13a56bc68e8e11470a5c8dfc3869ee530166f6261ea439" exitCode=60 Nov 03 15:45:02.900326 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.900223 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" event={"ID":"148ee23d-c6d4-42b6-9173-e16173fe1063","Type":"ContainerDied","Data":"2c142cea75e013781a13a56bc68e8e11470a5c8dfc3869ee530166f6261ea439"} Nov 03 15:45:02.900539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.900523 4557 scope.go:122] "RemoveContainer" containerID="2c142cea75e013781a13a56bc68e8e11470a5c8dfc3869ee530166f6261ea439" Nov 03 15:45:02.902013 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.901989 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq" event={"ID":"ce046a64-2b99-43e2-a354-a4129e45af72","Type":"ContainerStarted","Data":"bffb3e6fd991a0ba63f82d32cf32bd30ff5ceb11dce73ef3d42d9f02b2d166c3"} Nov 03 15:45:02.902114 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.902032 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq" event={"ID":"ce046a64-2b99-43e2-a354-a4129e45af72","Type":"ContainerStarted","Data":"19c19b06c697c14e80f4149d257006749215d620e7ead089963b83fb3721d57c"} Nov 03 15:45:02.904048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.903942 4557 generic.go:356] "Generic (PLEG): container finished" podID="a177f815-6766-447d-afea-c04c1a5dc18d" containerID="8de70d3a7480b1ded7a7ea1cbdec6f4468ed2a2776af55f64e32c3aca9a017c0" exitCode=60 Nov 03 15:45:02.904048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.904030 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" event={"ID":"a177f815-6766-447d-afea-c04c1a5dc18d","Type":"ContainerDied","Data":"8de70d3a7480b1ded7a7ea1cbdec6f4468ed2a2776af55f64e32c3aca9a017c0"} Nov 03 15:45:02.904292 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.904267 4557 scope.go:122] "RemoveContainer" containerID="8de70d3a7480b1ded7a7ea1cbdec6f4468ed2a2776af55f64e32c3aca9a017c0" Nov 03 15:45:02.905832 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.905813 4557 generic.go:356] "Generic (PLEG): container finished" podID="82aedd62-09e8-4bf9-aa75-fab98c1cf785" containerID="b41e2414ae3bf00715456e22116f861b488e3c20e8bee2c5402f818e9bfd1487" exitCode=60 Nov 03 15:45:02.905921 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.905894 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" event={"ID":"82aedd62-09e8-4bf9-aa75-fab98c1cf785","Type":"ContainerDied","Data":"b41e2414ae3bf00715456e22116f861b488e3c20e8bee2c5402f818e9bfd1487"} Nov 03 15:45:02.906083 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.906069 4557 scope.go:122] "RemoveContainer" containerID="b41e2414ae3bf00715456e22116f861b488e3c20e8bee2c5402f818e9bfd1487" Nov 03 15:45:02.907440 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.907421 4557 generic.go:356] "Generic (PLEG): container finished" podID="c6f53996-4386-4d7f-be27-4730e4804945" containerID="084c299025219540e6d7e37caa3ca709b8f1a86a8b0ddac274629ff48f4606ae" exitCode=60 Nov 03 15:45:02.907543 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.907487 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" event={"ID":"c6f53996-4386-4d7f-be27-4730e4804945","Type":"ContainerDied","Data":"084c299025219540e6d7e37caa3ca709b8f1a86a8b0ddac274629ff48f4606ae"} Nov 03 15:45:02.907678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.907663 4557 scope.go:122] "RemoveContainer" containerID="084c299025219540e6d7e37caa3ca709b8f1a86a8b0ddac274629ff48f4606ae" Nov 03 15:45:02.909170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.909150 4557 generic.go:356] "Generic (PLEG): container finished" podID="43aaa640-044f-43cd-ab66-089ed62e48fb" containerID="f0cf8c417ce1d115e41f8a7c51fb8d7210cd87ec2e8cb5cc5fb0bdc7e30eb650" exitCode=60 Nov 03 15:45:02.909251 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.909175 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" event={"ID":"43aaa640-044f-43cd-ab66-089ed62e48fb","Type":"ContainerDied","Data":"f0cf8c417ce1d115e41f8a7c51fb8d7210cd87ec2e8cb5cc5fb0bdc7e30eb650"} Nov 03 15:45:02.909430 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.909414 4557 scope.go:122] "RemoveContainer" containerID="f0cf8c417ce1d115e41f8a7c51fb8d7210cd87ec2e8cb5cc5fb0bdc7e30eb650" Nov 03 15:45:02.911030 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.910889 4557 generic.go:356] "Generic (PLEG): container finished" podID="6120a8ff-bed0-4512-bd06-09e7e917a26c" containerID="cb6eefe6da4d0273989e40cd7fa01004a5e56acfdadc021f6146e9a92f98e891" exitCode=60 Nov 03 15:45:02.911030 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.910919 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" event={"ID":"6120a8ff-bed0-4512-bd06-09e7e917a26c","Type":"ContainerDied","Data":"cb6eefe6da4d0273989e40cd7fa01004a5e56acfdadc021f6146e9a92f98e891"} Nov 03 15:45:02.911245 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:02.911151 4557 scope.go:122] "RemoveContainer" containerID="cb6eefe6da4d0273989e40cd7fa01004a5e56acfdadc021f6146e9a92f98e891" Nov 03 15:45:03.605049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:03.604981 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq" podStartSLOduration=2.6049680029999998 podStartE2EDuration="2.604968003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:03.604159251 +0000 UTC m=+2213.447280794" watchObservedRunningTime="2026-11-03 15:45:03.604968003 +0000 UTC m=+2213.448089544" Nov 03 15:45:03.919504 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:03.919461 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" event={"ID":"a177f815-6766-447d-afea-c04c1a5dc18d","Type":"ContainerStarted","Data":"d0d10f0c40b6239172aedd04b1ad6390d2d25f125c43a209a5223d8437145e02"} Nov 03 15:45:03.921527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:03.921503 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" event={"ID":"82aedd62-09e8-4bf9-aa75-fab98c1cf785","Type":"ContainerStarted","Data":"6d115a20c779c1930cf670c4f1f36e981f0f6ae7ff746fba98bda8ea4207ab7f"} Nov 03 15:45:03.923384 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:03.923360 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" event={"ID":"c6f53996-4386-4d7f-be27-4730e4804945","Type":"ContainerStarted","Data":"d97c6646db99705407706888acd164dc76c5f0fbc45e72e4fe8f0459a015a102"} Nov 03 15:45:03.925304 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:03.925278 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" event={"ID":"43aaa640-044f-43cd-ab66-089ed62e48fb","Type":"ContainerStarted","Data":"884c98a7d0eccab3908fc1f7972e10266bdbe48f3223f9b4cc53b5e63f3fd9b6"} Nov 03 15:45:03.927015 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:03.926989 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" event={"ID":"6120a8ff-bed0-4512-bd06-09e7e917a26c","Type":"ContainerStarted","Data":"c0f4c5a1a6674f1366b938baf1145213addbdafe7efa5ea5198bd22f30423c58"} Nov 03 15:45:03.928901 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:03.928874 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" event={"ID":"148ee23d-c6d4-42b6-9173-e16173fe1063","Type":"ContainerStarted","Data":"f8c4cec031ef580319acdb0141a9245822869470866d1df32f09458e608f97d1"} Nov 03 15:45:03.930216 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:03.930192 4557 generic.go:356] "Generic (PLEG): container finished" podID="ce046a64-2b99-43e2-a354-a4129e45af72" containerID="bffb3e6fd991a0ba63f82d32cf32bd30ff5ceb11dce73ef3d42d9f02b2d166c3" exitCode=0 Nov 03 15:45:03.930287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:03.930232 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq" event={"ID":"ce046a64-2b99-43e2-a354-a4129e45af72","Type":"ContainerDied","Data":"bffb3e6fd991a0ba63f82d32cf32bd30ff5ceb11dce73ef3d42d9f02b2d166c3"} Nov 03 15:45:04.938876 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.938837 4557 generic.go:356] "Generic (PLEG): container finished" podID="a177f815-6766-447d-afea-c04c1a5dc18d" containerID="d0d10f0c40b6239172aedd04b1ad6390d2d25f125c43a209a5223d8437145e02" exitCode=60 Nov 03 15:45:04.939335 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.938917 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" event={"ID":"a177f815-6766-447d-afea-c04c1a5dc18d","Type":"ContainerDied","Data":"d0d10f0c40b6239172aedd04b1ad6390d2d25f125c43a209a5223d8437145e02"} Nov 03 15:45:04.939335 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.938983 4557 scope.go:122] "RemoveContainer" containerID="8de70d3a7480b1ded7a7ea1cbdec6f4468ed2a2776af55f64e32c3aca9a017c0" Nov 03 15:45:04.939335 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.939187 4557 scope.go:122] "RemoveContainer" containerID="d0d10f0c40b6239172aedd04b1ad6390d2d25f125c43a209a5223d8437145e02" Nov 03 15:45:04.939557 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:04.939533 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw_odh-ai-gateway-infra(a177f815-6766-447d-afea-c04c1a5dc18d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" podUID="a177f815-6766-447d-afea-c04c1a5dc18d" Nov 03 15:45:04.941003 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.940982 4557 generic.go:356] "Generic (PLEG): container finished" podID="82aedd62-09e8-4bf9-aa75-fab98c1cf785" containerID="6d115a20c779c1930cf670c4f1f36e981f0f6ae7ff746fba98bda8ea4207ab7f" exitCode=60 Nov 03 15:45:04.941108 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.941018 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" event={"ID":"82aedd62-09e8-4bf9-aa75-fab98c1cf785","Type":"ContainerDied","Data":"6d115a20c779c1930cf670c4f1f36e981f0f6ae7ff746fba98bda8ea4207ab7f"} Nov 03 15:45:04.941271 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.941257 4557 scope.go:122] "RemoveContainer" containerID="6d115a20c779c1930cf670c4f1f36e981f0f6ae7ff746fba98bda8ea4207ab7f" Nov 03 15:45:04.941560 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:04.941535 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895345-4zwrs_odh-ai-gateway-infra(82aedd62-09e8-4bf9-aa75-fab98c1cf785)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" podUID="82aedd62-09e8-4bf9-aa75-fab98c1cf785" Nov 03 15:45:04.943037 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.943016 4557 generic.go:356] "Generic (PLEG): container finished" podID="c6f53996-4386-4d7f-be27-4730e4804945" containerID="d97c6646db99705407706888acd164dc76c5f0fbc45e72e4fe8f0459a015a102" exitCode=60 Nov 03 15:45:04.943110 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.943083 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" event={"ID":"c6f53996-4386-4d7f-be27-4730e4804945","Type":"ContainerDied","Data":"d97c6646db99705407706888acd164dc76c5f0fbc45e72e4fe8f0459a015a102"} Nov 03 15:45:04.943306 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.943288 4557 scope.go:122] "RemoveContainer" containerID="d97c6646db99705407706888acd164dc76c5f0fbc45e72e4fe8f0459a015a102" Nov 03 15:45:04.943573 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:04.943549 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc_odh-ai-gateway-infra(c6f53996-4386-4d7f-be27-4730e4804945)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" podUID="c6f53996-4386-4d7f-be27-4730e4804945" Nov 03 15:45:04.945249 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.945229 4557 generic.go:356] "Generic (PLEG): container finished" podID="43aaa640-044f-43cd-ab66-089ed62e48fb" containerID="884c98a7d0eccab3908fc1f7972e10266bdbe48f3223f9b4cc53b5e63f3fd9b6" exitCode=60 Nov 03 15:45:04.945362 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.945289 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" event={"ID":"43aaa640-044f-43cd-ab66-089ed62e48fb","Type":"ContainerDied","Data":"884c98a7d0eccab3908fc1f7972e10266bdbe48f3223f9b4cc53b5e63f3fd9b6"} Nov 03 15:45:04.945539 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.945523 4557 scope.go:122] "RemoveContainer" containerID="884c98a7d0eccab3908fc1f7972e10266bdbe48f3223f9b4cc53b5e63f3fd9b6" Nov 03 15:45:04.945821 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:04.945801 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k_odh-ai-gateway-infra(43aaa640-044f-43cd-ab66-089ed62e48fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" podUID="43aaa640-044f-43cd-ab66-089ed62e48fb" Nov 03 15:45:04.947277 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.947257 4557 generic.go:356] "Generic (PLEG): container finished" podID="6120a8ff-bed0-4512-bd06-09e7e917a26c" containerID="c0f4c5a1a6674f1366b938baf1145213addbdafe7efa5ea5198bd22f30423c58" exitCode=60 Nov 03 15:45:04.947364 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.947331 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" event={"ID":"6120a8ff-bed0-4512-bd06-09e7e917a26c","Type":"ContainerDied","Data":"c0f4c5a1a6674f1366b938baf1145213addbdafe7efa5ea5198bd22f30423c58"} Nov 03 15:45:04.947505 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.947486 4557 scope.go:122] "RemoveContainer" containerID="c0f4c5a1a6674f1366b938baf1145213addbdafe7efa5ea5198bd22f30423c58" Nov 03 15:45:04.947765 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:04.947744 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7_odh-ai-gateway-infra(6120a8ff-bed0-4512-bd06-09e7e917a26c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" podUID="6120a8ff-bed0-4512-bd06-09e7e917a26c" Nov 03 15:45:04.949393 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.949372 4557 generic.go:356] "Generic (PLEG): container finished" podID="148ee23d-c6d4-42b6-9173-e16173fe1063" containerID="f8c4cec031ef580319acdb0141a9245822869470866d1df32f09458e608f97d1" exitCode=60 Nov 03 15:45:04.949486 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.949458 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" event={"ID":"148ee23d-c6d4-42b6-9173-e16173fe1063","Type":"ContainerDied","Data":"f8c4cec031ef580319acdb0141a9245822869470866d1df32f09458e608f97d1"} Nov 03 15:45:04.949674 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.949657 4557 scope.go:122] "RemoveContainer" containerID="f8c4cec031ef580319acdb0141a9245822869470866d1df32f09458e608f97d1" Nov 03 15:45:04.949968 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:04.949871 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw_odh-ai-gateway-infra(148ee23d-c6d4-42b6-9173-e16173fe1063)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" podUID="148ee23d-c6d4-42b6-9173-e16173fe1063" Nov 03 15:45:04.952411 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.952392 4557 scope.go:122] "RemoveContainer" containerID="b41e2414ae3bf00715456e22116f861b488e3c20e8bee2c5402f818e9bfd1487" Nov 03 15:45:04.965262 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.965241 4557 scope.go:122] "RemoveContainer" containerID="084c299025219540e6d7e37caa3ca709b8f1a86a8b0ddac274629ff48f4606ae" Nov 03 15:45:04.978741 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.978219 4557 scope.go:122] "RemoveContainer" containerID="f0cf8c417ce1d115e41f8a7c51fb8d7210cd87ec2e8cb5cc5fb0bdc7e30eb650" Nov 03 15:45:04.992616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:04.992560 4557 scope.go:122] "RemoveContainer" containerID="cb6eefe6da4d0273989e40cd7fa01004a5e56acfdadc021f6146e9a92f98e891" Nov 03 15:45:05.007077 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.007048 4557 scope.go:122] "RemoveContainer" containerID="2c142cea75e013781a13a56bc68e8e11470a5c8dfc3869ee530166f6261ea439" Nov 03 15:45:05.103880 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.103855 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq" Nov 03 15:45:05.181687 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.181658 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce046a64-2b99-43e2-a354-a4129e45af72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce046a64-2b99-43e2-a354-a4129e45af72-maas-api-openshift-ca\") pod \"ce046a64-2b99-43e2-a354-a4129e45af72\" (UID: \"ce046a64-2b99-43e2-a354-a4129e45af72\") " Nov 03 15:45:05.181835 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.181795 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce046a64-2b99-43e2-a354-a4129e45af72-kube-api-access-pwr5v\" (UniqueName: \"kubernetes.io/projected/ce046a64-2b99-43e2-a354-a4129e45af72-kube-api-access-pwr5v\") pod \"ce046a64-2b99-43e2-a354-a4129e45af72\" (UID: \"ce046a64-2b99-43e2-a354-a4129e45af72\") " Nov 03 15:45:05.181970 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.181838 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce046a64-2b99-43e2-a354-a4129e45af72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce046a64-2b99-43e2-a354-a4129e45af72-maas-api-xks-ca\") pod \"ce046a64-2b99-43e2-a354-a4129e45af72\" (UID: \"ce046a64-2b99-43e2-a354-a4129e45af72\") " Nov 03 15:45:05.182057 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.182035 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce046a64-2b99-43e2-a354-a4129e45af72-maas-api-openshift-ca" pod "ce046a64-2b99-43e2-a354-a4129e45af72" (UID: "ce046a64-2b99-43e2-a354-a4129e45af72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:05.182674 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.182433 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce046a64-2b99-43e2-a354-a4129e45af72-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:05.184531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.184504 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce046a64-2b99-43e2-a354-a4129e45af72-maas-api-xks-ca" pod "ce046a64-2b99-43e2-a354-a4129e45af72" (UID: "ce046a64-2b99-43e2-a354-a4129e45af72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:45:05.184643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.184525 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce046a64-2b99-43e2-a354-a4129e45af72-kube-api-access-pwr5v" pod "ce046a64-2b99-43e2-a354-a4129e45af72" (UID: "ce046a64-2b99-43e2-a354-a4129e45af72"). InnerVolumeSpecName "kube-api-access-pwr5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:05.283645 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.283615 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwr5v\" (UniqueName: \"kubernetes.io/projected/ce046a64-2b99-43e2-a354-a4129e45af72-kube-api-access-pwr5v\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:05.283645 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.283644 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce046a64-2b99-43e2-a354-a4129e45af72-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:05.957918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.957890 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq" Nov 03 15:45:05.958317 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.957893 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq" event={"ID":"ce046a64-2b99-43e2-a354-a4129e45af72","Type":"ContainerDied","Data":"19c19b06c697c14e80f4149d257006749215d620e7ead089963b83fb3721d57c"} Nov 03 15:45:05.958317 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.958016 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19c19b06c697c14e80f4149d257006749215d620e7ead089963b83fb3721d57c" Nov 03 15:45:05.960349 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.960326 4557 scope.go:122] "RemoveContainer" containerID="d0d10f0c40b6239172aedd04b1ad6390d2d25f125c43a209a5223d8437145e02" Nov 03 15:45:05.960717 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:05.960693 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw_odh-ai-gateway-infra(a177f815-6766-447d-afea-c04c1a5dc18d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" podUID="a177f815-6766-447d-afea-c04c1a5dc18d" Nov 03 15:45:05.961807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.961784 4557 scope.go:122] "RemoveContainer" containerID="6d115a20c779c1930cf670c4f1f36e981f0f6ae7ff746fba98bda8ea4207ab7f" Nov 03 15:45:05.962077 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:05.962057 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895345-4zwrs_odh-ai-gateway-infra(82aedd62-09e8-4bf9-aa75-fab98c1cf785)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" podUID="82aedd62-09e8-4bf9-aa75-fab98c1cf785" Nov 03 15:45:05.963465 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.963447 4557 scope.go:122] "RemoveContainer" containerID="d97c6646db99705407706888acd164dc76c5f0fbc45e72e4fe8f0459a015a102" Nov 03 15:45:05.963691 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:05.963673 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc_odh-ai-gateway-infra(c6f53996-4386-4d7f-be27-4730e4804945)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" podUID="c6f53996-4386-4d7f-be27-4730e4804945" Nov 03 15:45:05.965017 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.965001 4557 scope.go:122] "RemoveContainer" containerID="884c98a7d0eccab3908fc1f7972e10266bdbe48f3223f9b4cc53b5e63f3fd9b6" Nov 03 15:45:05.965238 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:05.965216 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k_odh-ai-gateway-infra(43aaa640-044f-43cd-ab66-089ed62e48fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" podUID="43aaa640-044f-43cd-ab66-089ed62e48fb" Nov 03 15:45:05.966365 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.966347 4557 scope.go:122] "RemoveContainer" containerID="c0f4c5a1a6674f1366b938baf1145213addbdafe7efa5ea5198bd22f30423c58" Nov 03 15:45:05.966564 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:05.966543 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7_odh-ai-gateway-infra(6120a8ff-bed0-4512-bd06-09e7e917a26c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" podUID="6120a8ff-bed0-4512-bd06-09e7e917a26c" Nov 03 15:45:05.967825 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:05.967807 4557 scope.go:122] "RemoveContainer" containerID="f8c4cec031ef580319acdb0141a9245822869470866d1df32f09458e608f97d1" Nov 03 15:45:05.968001 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:05.967986 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw_odh-ai-gateway-infra(148ee23d-c6d4-42b6-9173-e16173fe1063)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" podUID="148ee23d-c6d4-42b6-9173-e16173fe1063" Nov 03 15:45:08.298306 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:08.297331 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:08.496169 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:08.496134 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:12.634353 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:12.634317 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:12.821895 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:12.821847 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:13.933528 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:13.933488 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:16.279134 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:16.279105 4557 scope.go:122] "RemoveContainer" containerID="f8c4cec031ef580319acdb0141a9245822869470866d1df32f09458e608f97d1" Nov 03 15:45:16.529549 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:16.529475 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:45:16.849251 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:16.849222 4557 scope.go:122] "RemoveContainer" containerID="f8880f52e865663f697df9ccd5d3b5a3859b11b8c75a24497ef2dd829434e589" Nov 03 15:45:16.858319 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:16.858301 4557 scope.go:122] "RemoveContainer" containerID="05c3df4df09339a9600e716e03364b9739214e63efa0503418636d7c1c5b9e9f" Nov 03 15:45:16.866377 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:16.866357 4557 scope.go:122] "RemoveContainer" containerID="38e20ce13dcd6b7041942ec344abcb73aece1560c73234ebfca46366bfe7c8f0" Nov 03 15:45:17.036092 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:17.036053 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" event={"ID":"148ee23d-c6d4-42b6-9173-e16173fe1063","Type":"ContainerStarted","Data":"b17898ff9d1b7f08160d661e1f32c4b5e9ba12b05fca6f0970f16b69b29e6e95"} Nov 03 15:45:17.279480 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:17.279451 4557 scope.go:122] "RemoveContainer" containerID="6d115a20c779c1930cf670c4f1f36e981f0f6ae7ff746fba98bda8ea4207ab7f" Nov 03 15:45:17.279883 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:17.279561 4557 scope.go:122] "RemoveContainer" containerID="d97c6646db99705407706888acd164dc76c5f0fbc45e72e4fe8f0459a015a102" Nov 03 15:45:17.279883 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:17.279664 4557 scope.go:122] "RemoveContainer" containerID="c0f4c5a1a6674f1366b938baf1145213addbdafe7efa5ea5198bd22f30423c58" Nov 03 15:45:18.043337 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:18.043251 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" event={"ID":"82aedd62-09e8-4bf9-aa75-fab98c1cf785","Type":"ContainerStarted","Data":"a9b64427515db16a5756f66742716a24631fdfaa4a4c3cba0adaee95002d4485"} Nov 03 15:45:18.045281 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:18.045250 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" event={"ID":"c6f53996-4386-4d7f-be27-4730e4804945","Type":"ContainerStarted","Data":"606ae1c1eecc3f612f46426f768228be8f3da1d1d9ca9af5dffd67bc9744b886"} Nov 03 15:45:18.047174 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:18.047151 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" event={"ID":"6120a8ff-bed0-4512-bd06-09e7e917a26c","Type":"ContainerStarted","Data":"671a51fd606dbaa2d9441bcb15dd5fdaa7a66d7e6f35503bd21565a49e0b3ab8"} Nov 03 15:45:18.049085 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:18.049061 4557 generic.go:356] "Generic (PLEG): container finished" podID="148ee23d-c6d4-42b6-9173-e16173fe1063" containerID="b17898ff9d1b7f08160d661e1f32c4b5e9ba12b05fca6f0970f16b69b29e6e95" exitCode=60 Nov 03 15:45:18.049192 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:18.049123 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" event={"ID":"148ee23d-c6d4-42b6-9173-e16173fe1063","Type":"ContainerDied","Data":"b17898ff9d1b7f08160d661e1f32c4b5e9ba12b05fca6f0970f16b69b29e6e95"} Nov 03 15:45:18.049192 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:18.049176 4557 scope.go:122] "RemoveContainer" containerID="f8c4cec031ef580319acdb0141a9245822869470866d1df32f09458e608f97d1" Nov 03 15:45:18.049468 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:18.049440 4557 scope.go:122] "RemoveContainer" containerID="b17898ff9d1b7f08160d661e1f32c4b5e9ba12b05fca6f0970f16b69b29e6e95" Nov 03 15:45:18.049751 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:18.049727 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw_odh-ai-gateway-infra(148ee23d-c6d4-42b6-9173-e16173fe1063)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" podUID="148ee23d-c6d4-42b6-9173-e16173fe1063" Nov 03 15:45:18.174315 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:18.174280 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw"] Nov 03 15:45:18.278319 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:18.278290 4557 scope.go:122] "RemoveContainer" containerID="884c98a7d0eccab3908fc1f7972e10266bdbe48f3223f9b4cc53b5e63f3fd9b6" Nov 03 15:45:18.278534 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:18.278407 4557 scope.go:122] "RemoveContainer" containerID="d0d10f0c40b6239172aedd04b1ad6390d2d25f125c43a209a5223d8437145e02" Nov 03 15:45:18.575686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:18.575608 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:19.056271 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.056238 4557 generic.go:356] "Generic (PLEG): container finished" podID="c6f53996-4386-4d7f-be27-4730e4804945" containerID="606ae1c1eecc3f612f46426f768228be8f3da1d1d9ca9af5dffd67bc9744b886" exitCode=60 Nov 03 15:45:19.056448 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.056266 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" event={"ID":"c6f53996-4386-4d7f-be27-4730e4804945","Type":"ContainerDied","Data":"606ae1c1eecc3f612f46426f768228be8f3da1d1d9ca9af5dffd67bc9744b886"} Nov 03 15:45:19.056448 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.056319 4557 scope.go:122] "RemoveContainer" containerID="d97c6646db99705407706888acd164dc76c5f0fbc45e72e4fe8f0459a015a102" Nov 03 15:45:19.056707 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.056687 4557 scope.go:122] "RemoveContainer" containerID="606ae1c1eecc3f612f46426f768228be8f3da1d1d9ca9af5dffd67bc9744b886" Nov 03 15:45:19.057053 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:19.057014 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc_odh-ai-gateway-infra(c6f53996-4386-4d7f-be27-4730e4804945)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" podUID="c6f53996-4386-4d7f-be27-4730e4804945" Nov 03 15:45:19.058281 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.058261 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" event={"ID":"43aaa640-044f-43cd-ab66-089ed62e48fb","Type":"ContainerStarted","Data":"8e564182840490cff31b3e6ff4967bae59e84eb90bf7897674a76208fc67678a"} Nov 03 15:45:19.060100 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.060083 4557 generic.go:356] "Generic (PLEG): container finished" podID="6120a8ff-bed0-4512-bd06-09e7e917a26c" containerID="671a51fd606dbaa2d9441bcb15dd5fdaa7a66d7e6f35503bd21565a49e0b3ab8" exitCode=60 Nov 03 15:45:19.060191 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.060139 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" event={"ID":"6120a8ff-bed0-4512-bd06-09e7e917a26c","Type":"ContainerDied","Data":"671a51fd606dbaa2d9441bcb15dd5fdaa7a66d7e6f35503bd21565a49e0b3ab8"} Nov 03 15:45:19.060515 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.060498 4557 scope.go:122] "RemoveContainer" containerID="671a51fd606dbaa2d9441bcb15dd5fdaa7a66d7e6f35503bd21565a49e0b3ab8" Nov 03 15:45:19.060771 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:19.060746 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7_odh-ai-gateway-infra(6120a8ff-bed0-4512-bd06-09e7e917a26c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" podUID="6120a8ff-bed0-4512-bd06-09e7e917a26c" Nov 03 15:45:19.063740 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.063718 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" event={"ID":"a177f815-6766-447d-afea-c04c1a5dc18d","Type":"ContainerStarted","Data":"78fc227f57af5a308bec95d6b8346ac87e497248470602d3e072372f708f2379"} Nov 03 15:45:19.065849 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.065829 4557 generic.go:356] "Generic (PLEG): container finished" podID="82aedd62-09e8-4bf9-aa75-fab98c1cf785" containerID="a9b64427515db16a5756f66742716a24631fdfaa4a4c3cba0adaee95002d4485" exitCode=60 Nov 03 15:45:19.065954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.065864 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" event={"ID":"82aedd62-09e8-4bf9-aa75-fab98c1cf785","Type":"ContainerDied","Data":"a9b64427515db16a5756f66742716a24631fdfaa4a4c3cba0adaee95002d4485"} Nov 03 15:45:19.066115 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.066099 4557 scope.go:122] "RemoveContainer" containerID="a9b64427515db16a5756f66742716a24631fdfaa4a4c3cba0adaee95002d4485" Nov 03 15:45:19.066289 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:19.066271 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895345-4zwrs_odh-ai-gateway-infra(82aedd62-09e8-4bf9-aa75-fab98c1cf785)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" podUID="82aedd62-09e8-4bf9-aa75-fab98c1cf785" Nov 03 15:45:19.069362 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.069341 4557 scope.go:122] "RemoveContainer" containerID="c0f4c5a1a6674f1366b938baf1145213addbdafe7efa5ea5198bd22f30423c58" Nov 03 15:45:19.151636 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.151571 4557 scope.go:122] "RemoveContainer" containerID="6d115a20c779c1930cf670c4f1f36e981f0f6ae7ff746fba98bda8ea4207ab7f" Nov 03 15:45:19.174077 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.174053 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs"] Nov 03 15:45:19.279869 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.279841 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" Nov 03 15:45:19.298073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.298047 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc"] Nov 03 15:45:19.307123 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.307068 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/148ee23d-c6d4-42b6-9173-e16173fe1063-kube-api-access-x9xxw\" (UniqueName: \"kubernetes.io/projected/148ee23d-c6d4-42b6-9173-e16173fe1063-kube-api-access-x9xxw\") pod \"148ee23d-c6d4-42b6-9173-e16173fe1063\" (UID: \"148ee23d-c6d4-42b6-9173-e16173fe1063\") " Nov 03 15:45:19.307123 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.307097 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/148ee23d-c6d4-42b6-9173-e16173fe1063-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/148ee23d-c6d4-42b6-9173-e16173fe1063-maas-api-service-ca\") pod \"148ee23d-c6d4-42b6-9173-e16173fe1063\" (UID: \"148ee23d-c6d4-42b6-9173-e16173fe1063\") " Nov 03 15:45:19.307557 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.307526 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/148ee23d-c6d4-42b6-9173-e16173fe1063-maas-api-service-ca" pod "148ee23d-c6d4-42b6-9173-e16173fe1063" (UID: "148ee23d-c6d4-42b6-9173-e16173fe1063"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:19.377438 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.377070 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148ee23d-c6d4-42b6-9173-e16173fe1063-kube-api-access-x9xxw" pod "148ee23d-c6d4-42b6-9173-e16173fe1063" (UID: "148ee23d-c6d4-42b6-9173-e16173fe1063"). InnerVolumeSpecName "kube-api-access-x9xxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:19.409011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.408976 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9xxw\" (UniqueName: \"kubernetes.io/projected/148ee23d-c6d4-42b6-9173-e16173fe1063-kube-api-access-x9xxw\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:19.409011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.409012 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/148ee23d-c6d4-42b6-9173-e16173fe1063-maas-api-service-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:19.561046 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:19.560953 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7"] Nov 03 15:45:20.078751 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.078711 4557 generic.go:356] "Generic (PLEG): container finished" podID="43aaa640-044f-43cd-ab66-089ed62e48fb" containerID="8e564182840490cff31b3e6ff4967bae59e84eb90bf7897674a76208fc67678a" exitCode=60 Nov 03 15:45:20.079131 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.078778 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" event={"ID":"43aaa640-044f-43cd-ab66-089ed62e48fb","Type":"ContainerDied","Data":"8e564182840490cff31b3e6ff4967bae59e84eb90bf7897674a76208fc67678a"} Nov 03 15:45:20.079131 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.078868 4557 scope.go:122] "RemoveContainer" containerID="884c98a7d0eccab3908fc1f7972e10266bdbe48f3223f9b4cc53b5e63f3fd9b6" Nov 03 15:45:20.079293 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.079262 4557 scope.go:122] "RemoveContainer" containerID="8e564182840490cff31b3e6ff4967bae59e84eb90bf7897674a76208fc67678a" Nov 03 15:45:20.079638 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:20.079605 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k_odh-ai-gateway-infra(43aaa640-044f-43cd-ab66-089ed62e48fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" podUID="43aaa640-044f-43cd-ab66-089ed62e48fb" Nov 03 15:45:20.081992 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.081975 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" Nov 03 15:45:20.082084 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.082004 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw" event={"ID":"148ee23d-c6d4-42b6-9173-e16173fe1063","Type":"ContainerDied","Data":"1d6ea5c8599930e8eed54b7606ab3d7c8005235dd66bf20252d3258c74def14b"} Nov 03 15:45:20.084327 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.084306 4557 generic.go:356] "Generic (PLEG): container finished" podID="a177f815-6766-447d-afea-c04c1a5dc18d" containerID="78fc227f57af5a308bec95d6b8346ac87e497248470602d3e072372f708f2379" exitCode=60 Nov 03 15:45:20.084446 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.084346 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" event={"ID":"a177f815-6766-447d-afea-c04c1a5dc18d","Type":"ContainerDied","Data":"78fc227f57af5a308bec95d6b8346ac87e497248470602d3e072372f708f2379"} Nov 03 15:45:20.084702 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.084686 4557 scope.go:122] "RemoveContainer" containerID="78fc227f57af5a308bec95d6b8346ac87e497248470602d3e072372f708f2379" Nov 03 15:45:20.084935 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:20.084915 4557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw_odh-ai-gateway-infra(a177f815-6766-447d-afea-c04c1a5dc18d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" podUID="a177f815-6766-447d-afea-c04c1a5dc18d" Nov 03 15:45:20.102868 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.102776 4557 scope.go:122] "RemoveContainer" containerID="b17898ff9d1b7f08160d661e1f32c4b5e9ba12b05fca6f0970f16b69b29e6e95" Nov 03 15:45:20.114763 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.114735 4557 scope.go:122] "RemoveContainer" containerID="d0d10f0c40b6239172aedd04b1ad6390d2d25f125c43a209a5223d8437145e02" Nov 03 15:45:20.231737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.231710 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:20.264374 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.264348 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" Nov 03 15:45:20.283694 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.283667 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" Nov 03 15:45:20.308501 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.308475 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" Nov 03 15:45:20.308641 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.308563 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw"] Nov 03 15:45:20.314314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.314293 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f356bb-29895345-7nhrw"] Nov 03 15:45:20.321649 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.321628 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k"] Nov 03 15:45:20.330777 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.330720 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f53996-4386-4d7f-be27-4730e4804945-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f53996-4386-4d7f-be27-4730e4804945-maas-api-service-ca\") pod \"c6f53996-4386-4d7f-be27-4730e4804945\" (UID: \"c6f53996-4386-4d7f-be27-4730e4804945\") " Nov 03 15:45:20.330777 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.330750 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f53996-4386-4d7f-be27-4730e4804945-kube-api-access-hqzj4\" (UniqueName: \"kubernetes.io/projected/c6f53996-4386-4d7f-be27-4730e4804945-kube-api-access-hqzj4\") pod \"c6f53996-4386-4d7f-be27-4730e4804945\" (UID: \"c6f53996-4386-4d7f-be27-4730e4804945\") " Nov 03 15:45:20.330946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.330841 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82aedd62-09e8-4bf9-aa75-fab98c1cf785-kube-api-access-x86kb\" (UniqueName: \"kubernetes.io/projected/82aedd62-09e8-4bf9-aa75-fab98c1cf785-kube-api-access-x86kb\") pod \"82aedd62-09e8-4bf9-aa75-fab98c1cf785\" (UID: \"82aedd62-09e8-4bf9-aa75-fab98c1cf785\") " Nov 03 15:45:20.330946 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.330887 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82aedd62-09e8-4bf9-aa75-fab98c1cf785-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aedd62-09e8-4bf9-aa75-fab98c1cf785-maas-api-service-ca\") pod \"82aedd62-09e8-4bf9-aa75-fab98c1cf785\" (UID: \"82aedd62-09e8-4bf9-aa75-fab98c1cf785\") " Nov 03 15:45:20.331058 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.330979 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6120a8ff-bed0-4512-bd06-09e7e917a26c-kube-api-access-fzp2s\" (UniqueName: \"kubernetes.io/projected/6120a8ff-bed0-4512-bd06-09e7e917a26c-kube-api-access-fzp2s\") pod \"6120a8ff-bed0-4512-bd06-09e7e917a26c\" (UID: \"6120a8ff-bed0-4512-bd06-09e7e917a26c\") " Nov 03 15:45:20.331116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.331060 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6120a8ff-bed0-4512-bd06-09e7e917a26c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6120a8ff-bed0-4512-bd06-09e7e917a26c-maas-api-service-ca\") pod \"6120a8ff-bed0-4512-bd06-09e7e917a26c\" (UID: \"6120a8ff-bed0-4512-bd06-09e7e917a26c\") " Nov 03 15:45:20.331201 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.331138 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f53996-4386-4d7f-be27-4730e4804945-maas-api-service-ca" pod "c6f53996-4386-4d7f-be27-4730e4804945" (UID: "c6f53996-4386-4d7f-be27-4730e4804945"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:20.331344 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.331316 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82aedd62-09e8-4bf9-aa75-fab98c1cf785-maas-api-service-ca" pod "82aedd62-09e8-4bf9-aa75-fab98c1cf785" (UID: "82aedd62-09e8-4bf9-aa75-fab98c1cf785"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:20.331552 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.331502 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6120a8ff-bed0-4512-bd06-09e7e917a26c-maas-api-service-ca" pod "6120a8ff-bed0-4512-bd06-09e7e917a26c" (UID: "6120a8ff-bed0-4512-bd06-09e7e917a26c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:20.331651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.331625 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aedd62-09e8-4bf9-aa75-fab98c1cf785-maas-api-service-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:20.331651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.331639 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6120a8ff-bed0-4512-bd06-09e7e917a26c-maas-api-service-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:20.331651 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.331648 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f53996-4386-4d7f-be27-4730e4804945-maas-api-service-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:20.333033 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.333012 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82aedd62-09e8-4bf9-aa75-fab98c1cf785-kube-api-access-x86kb" pod "82aedd62-09e8-4bf9-aa75-fab98c1cf785" (UID: "82aedd62-09e8-4bf9-aa75-fab98c1cf785"). InnerVolumeSpecName "kube-api-access-x86kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:20.333146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.333125 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f53996-4386-4d7f-be27-4730e4804945-kube-api-access-hqzj4" pod "c6f53996-4386-4d7f-be27-4730e4804945" (UID: "c6f53996-4386-4d7f-be27-4730e4804945"). InnerVolumeSpecName "kube-api-access-hqzj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:20.333319 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.333304 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6120a8ff-bed0-4512-bd06-09e7e917a26c-kube-api-access-fzp2s" pod "6120a8ff-bed0-4512-bd06-09e7e917a26c" (UID: "6120a8ff-bed0-4512-bd06-09e7e917a26c"). InnerVolumeSpecName "kube-api-access-fzp2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:20.432758 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.432722 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqzj4\" (UniqueName: \"kubernetes.io/projected/c6f53996-4386-4d7f-be27-4730e4804945-kube-api-access-hqzj4\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:20.432758 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.432752 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x86kb\" (UniqueName: \"kubernetes.io/projected/82aedd62-09e8-4bf9-aa75-fab98c1cf785-kube-api-access-x86kb\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:20.432758 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.432762 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzp2s\" (UniqueName: \"kubernetes.io/projected/6120a8ff-bed0-4512-bd06-09e7e917a26c-kube-api-access-fzp2s\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:20.517152 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.517115 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:20.556646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.556615 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw"] Nov 03 15:45:20.714073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:20.713989 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:21.092310 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.092281 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" Nov 03 15:45:21.092768 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.092281 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc" event={"ID":"c6f53996-4386-4d7f-be27-4730e4804945","Type":"ContainerDied","Data":"7e8811cf2583e21e2be03669a7b202312a188c2f71263e389bd3df7031575727"} Nov 03 15:45:21.092768 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.092411 4557 scope.go:122] "RemoveContainer" containerID="606ae1c1eecc3f612f46426f768228be8f3da1d1d9ca9af5dffd67bc9744b886" Nov 03 15:45:21.095550 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.095527 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" event={"ID":"6120a8ff-bed0-4512-bd06-09e7e917a26c","Type":"ContainerDied","Data":"e2205b87d06b49bf8bf3ea01529a85a282db5b1e1e0fed60cb2386a364bbef28"} Nov 03 15:45:21.095695 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.095534 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7" Nov 03 15:45:21.100865 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.100829 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" event={"ID":"82aedd62-09e8-4bf9-aa75-fab98c1cf785","Type":"ContainerDied","Data":"c13ebb756b283c37c267cf985ae7c95139bc91b1466950d2e6edbb5ff9700e62"} Nov 03 15:45:21.100865 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.100855 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs" Nov 03 15:45:21.104452 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.104434 4557 scope.go:122] "RemoveContainer" containerID="671a51fd606dbaa2d9441bcb15dd5fdaa7a66d7e6f35503bd21565a49e0b3ab8" Nov 03 15:45:21.116410 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.116376 4557 scope.go:122] "RemoveContainer" containerID="a9b64427515db16a5756f66742716a24631fdfaa4a4c3cba0adaee95002d4485" Nov 03 15:45:21.263398 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.263367 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:21.268886 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.268866 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" Nov 03 15:45:21.272473 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.272448 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" Nov 03 15:45:21.351777 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.351696 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a177f815-6766-447d-afea-c04c1a5dc18d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a177f815-6766-447d-afea-c04c1a5dc18d-maas-api-service-ca\") pod \"a177f815-6766-447d-afea-c04c1a5dc18d\" (UID: \"a177f815-6766-447d-afea-c04c1a5dc18d\") " Nov 03 15:45:21.351777 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.351766 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a177f815-6766-447d-afea-c04c1a5dc18d-kube-api-access-4lkcn\" (UniqueName: \"kubernetes.io/projected/a177f815-6766-447d-afea-c04c1a5dc18d-kube-api-access-4lkcn\") pod \"a177f815-6766-447d-afea-c04c1a5dc18d\" (UID: \"a177f815-6766-447d-afea-c04c1a5dc18d\") " Nov 03 15:45:21.352031 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.351882 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43aaa640-044f-43cd-ab66-089ed62e48fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43aaa640-044f-43cd-ab66-089ed62e48fb-maas-api-service-ca\") pod \"43aaa640-044f-43cd-ab66-089ed62e48fb\" (UID: \"43aaa640-044f-43cd-ab66-089ed62e48fb\") " Nov 03 15:45:21.352031 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.351926 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43aaa640-044f-43cd-ab66-089ed62e48fb-kube-api-access-kvpkk\" (UniqueName: \"kubernetes.io/projected/43aaa640-044f-43cd-ab66-089ed62e48fb-kube-api-access-kvpkk\") pod \"43aaa640-044f-43cd-ab66-089ed62e48fb\" (UID: \"43aaa640-044f-43cd-ab66-089ed62e48fb\") " Nov 03 15:45:21.352198 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.352175 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a177f815-6766-447d-afea-c04c1a5dc18d-maas-api-service-ca" pod "a177f815-6766-447d-afea-c04c1a5dc18d" (UID: "a177f815-6766-447d-afea-c04c1a5dc18d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:21.352272 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.352249 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43aaa640-044f-43cd-ab66-089ed62e48fb-maas-api-service-ca" pod "43aaa640-044f-43cd-ab66-089ed62e48fb" (UID: "43aaa640-044f-43cd-ab66-089ed62e48fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:21.352477 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.352462 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a177f815-6766-447d-afea-c04c1a5dc18d-maas-api-service-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:21.352513 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.352479 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43aaa640-044f-43cd-ab66-089ed62e48fb-maas-api-service-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:21.353884 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.353856 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a177f815-6766-447d-afea-c04c1a5dc18d-kube-api-access-4lkcn" pod "a177f815-6766-447d-afea-c04c1a5dc18d" (UID: "a177f815-6766-447d-afea-c04c1a5dc18d"). InnerVolumeSpecName "kube-api-access-4lkcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:21.353993 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.353913 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43aaa640-044f-43cd-ab66-089ed62e48fb-kube-api-access-kvpkk" pod "43aaa640-044f-43cd-ab66-089ed62e48fb" (UID: "43aaa640-044f-43cd-ab66-089ed62e48fb"). InnerVolumeSpecName "kube-api-access-kvpkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:21.404131 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.404093 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs"] Nov 03 15:45:21.409963 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.409937 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895345-4zwrs"] Nov 03 15:45:21.453975 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.453945 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lkcn\" (UniqueName: \"kubernetes.io/projected/a177f815-6766-447d-afea-c04c1a5dc18d-kube-api-access-4lkcn\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:21.453975 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.453971 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvpkk\" (UniqueName: \"kubernetes.io/projected/43aaa640-044f-43cd-ab66-089ed62e48fb-kube-api-access-kvpkk\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:21.520554 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.520516 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc"] Nov 03 15:45:21.537640 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.530946 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e62ff3-29895345-rnqdc"] Nov 03 15:45:21.757088 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.756988 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7"] Nov 03 15:45:21.762980 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:21.762946 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b841a-29895345-xxrw7"] Nov 03 15:45:22.107847 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:22.107804 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" Nov 03 15:45:22.108331 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:22.107800 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw" event={"ID":"a177f815-6766-447d-afea-c04c1a5dc18d","Type":"ContainerDied","Data":"7241a1727f9340ce243ab5c0bda94ea21a5d3ff46db110f8851386bb5a20852d"} Nov 03 15:45:22.108331 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:22.108011 4557 scope.go:122] "RemoveContainer" containerID="78fc227f57af5a308bec95d6b8346ac87e497248470602d3e072372f708f2379" Nov 03 15:45:22.111347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:22.111318 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" event={"ID":"43aaa640-044f-43cd-ab66-089ed62e48fb","Type":"ContainerDied","Data":"2280dc36a8cd41a89b1b4d8089979c1c75fdbb323fa18a2f62c3210d1579d97a"} Nov 03 15:45:22.111477 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:22.111360 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k" Nov 03 15:45:22.118227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:22.118209 4557 scope.go:122] "RemoveContainer" containerID="8e564182840490cff31b3e6ff4967bae59e84eb90bf7897674a76208fc67678a" Nov 03 15:45:22.274641 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:22.274601 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw"] Nov 03 15:45:22.285806 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:22.285773 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="148ee23d-c6d4-42b6-9173-e16173fe1063" path="/var/lib/kubelet/pods/148ee23d-c6d4-42b6-9173-e16173fe1063/volumes" Nov 03 15:45:22.286284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:22.286263 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6120a8ff-bed0-4512-bd06-09e7e917a26c" path="/var/lib/kubelet/pods/6120a8ff-bed0-4512-bd06-09e7e917a26c/volumes" Nov 03 15:45:22.286743 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:22.286725 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82aedd62-09e8-4bf9-aa75-fab98c1cf785" path="/var/lib/kubelet/pods/82aedd62-09e8-4bf9-aa75-fab98c1cf785/volumes" Nov 03 15:45:22.287440 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:22.287420 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f53996-4386-4d7f-be27-4730e4804945" path="/var/lib/kubelet/pods/c6f53996-4386-4d7f-be27-4730e4804945/volumes" Nov 03 15:45:22.287775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:22.287759 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f59d26-29895345-cwmhw"] Nov 03 15:45:22.389989 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:22.389910 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k"] Nov 03 15:45:22.394369 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:22.394346 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bc1f07-29895345-79j7k"] Nov 03 15:45:24.285825 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:24.285797 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43aaa640-044f-43cd-ab66-089ed62e48fb" path="/var/lib/kubelet/pods/43aaa640-044f-43cd-ab66-089ed62e48fb/volumes" Nov 03 15:45:24.286282 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:24.286170 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a177f815-6766-447d-afea-c04c1a5dc18d" path="/var/lib/kubelet/pods/a177f815-6766-447d-afea-c04c1a5dc18d/volumes" Nov 03 15:45:25.378663 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:25.378632 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:25.838661 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:25.838621 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:27.397136 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:27.397103 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:27.583073 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:27.583036 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:27.768891 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:27.768805 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:28.612982 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.612947 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb"] Nov 03 15:45:28.613460 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613446 4557 cpu_manager.go:414] "containerMap: removing container" podUID="82aedd62-09e8-4bf9-aa75-fab98c1cf785" containerName="cleanup" Nov 03 15:45:28.613510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613461 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82aedd62-09e8-4bf9-aa75-fab98c1cf785" containerName="cleanup" Nov 03 15:45:28.613510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613472 4557 cpu_manager.go:414] "containerMap: removing container" podUID="6120a8ff-bed0-4512-bd06-09e7e917a26c" containerName="cleanup" Nov 03 15:45:28.613510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613477 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6120a8ff-bed0-4512-bd06-09e7e917a26c" containerName="cleanup" Nov 03 15:45:28.613510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613486 4557 cpu_manager.go:414] "containerMap: removing container" podUID="43aaa640-044f-43cd-ab66-089ed62e48fb" containerName="cleanup" Nov 03 15:45:28.613510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613491 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43aaa640-044f-43cd-ab66-089ed62e48fb" containerName="cleanup" Nov 03 15:45:28.613510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613499 4557 cpu_manager.go:414] "containerMap: removing container" podUID="6120a8ff-bed0-4512-bd06-09e7e917a26c" containerName="cleanup" Nov 03 15:45:28.613510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613504 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6120a8ff-bed0-4512-bd06-09e7e917a26c" containerName="cleanup" Nov 03 15:45:28.613510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613513 4557 cpu_manager.go:414] "containerMap: removing container" podUID="c6f53996-4386-4d7f-be27-4730e4804945" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613518 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f53996-4386-4d7f-be27-4730e4804945" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613525 4557 cpu_manager.go:414] "containerMap: removing container" podUID="43aaa640-044f-43cd-ab66-089ed62e48fb" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613529 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43aaa640-044f-43cd-ab66-089ed62e48fb" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613538 4557 cpu_manager.go:414] "containerMap: removing container" podUID="a177f815-6766-447d-afea-c04c1a5dc18d" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613543 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a177f815-6766-447d-afea-c04c1a5dc18d" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613554 4557 cpu_manager.go:414] "containerMap: removing container" podUID="6120a8ff-bed0-4512-bd06-09e7e917a26c" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613560 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6120a8ff-bed0-4512-bd06-09e7e917a26c" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613567 4557 cpu_manager.go:414] "containerMap: removing container" podUID="a177f815-6766-447d-afea-c04c1a5dc18d" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613571 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a177f815-6766-447d-afea-c04c1a5dc18d" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613590 4557 cpu_manager.go:414] "containerMap: removing container" podUID="c6f53996-4386-4d7f-be27-4730e4804945" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613595 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f53996-4386-4d7f-be27-4730e4804945" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613605 4557 cpu_manager.go:414] "containerMap: removing container" podUID="82aedd62-09e8-4bf9-aa75-fab98c1cf785" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613609 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82aedd62-09e8-4bf9-aa75-fab98c1cf785" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613617 4557 cpu_manager.go:414] "containerMap: removing container" podUID="a177f815-6766-447d-afea-c04c1a5dc18d" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613621 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a177f815-6766-447d-afea-c04c1a5dc18d" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613628 4557 cpu_manager.go:414] "containerMap: removing container" podUID="148ee23d-c6d4-42b6-9173-e16173fe1063" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613632 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="148ee23d-c6d4-42b6-9173-e16173fe1063" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613640 4557 cpu_manager.go:414] "containerMap: removing container" podUID="ce046a64-2b99-43e2-a354-a4129e45af72" containerName="revoke-keys" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613645 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce046a64-2b99-43e2-a354-a4129e45af72" containerName="revoke-keys" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613652 4557 cpu_manager.go:414] "containerMap: removing container" podUID="148ee23d-c6d4-42b6-9173-e16173fe1063" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613656 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="148ee23d-c6d4-42b6-9173-e16173fe1063" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613735 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="82aedd62-09e8-4bf9-aa75-fab98c1cf785" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613743 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="a177f815-6766-447d-afea-c04c1a5dc18d" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613749 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="148ee23d-c6d4-42b6-9173-e16173fe1063" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613756 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce046a64-2b99-43e2-a354-a4129e45af72" containerName="revoke-keys" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613761 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="148ee23d-c6d4-42b6-9173-e16173fe1063" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613767 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="43aaa640-044f-43cd-ab66-089ed62e48fb" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613774 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f53996-4386-4d7f-be27-4730e4804945" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613778 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="148ee23d-c6d4-42b6-9173-e16173fe1063" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613784 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="82aedd62-09e8-4bf9-aa75-fab98c1cf785" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613788 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="6120a8ff-bed0-4512-bd06-09e7e917a26c" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613792 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="43aaa640-044f-43cd-ab66-089ed62e48fb" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613798 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="6120a8ff-bed0-4512-bd06-09e7e917a26c" containerName="cleanup" Nov 03 15:45:28.613790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613804 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="43aaa640-044f-43cd-ab66-089ed62e48fb" containerName="cleanup" Nov 03 15:45:28.615134 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613810 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="a177f815-6766-447d-afea-c04c1a5dc18d" containerName="cleanup" Nov 03 15:45:28.615134 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.613814 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f53996-4386-4d7f-be27-4730e4804945" containerName="cleanup" Nov 03 15:45:28.618728 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.618710 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb" Nov 03 15:45:28.674869 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.674841 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb"] Nov 03 15:45:28.721443 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.721415 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:45:28.721610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.721497 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:45:28.755982 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.755949 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb\" (UID: \"95bc0b39-5736-442b-96ab-d9e6ea6d63b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb" Nov 03 15:45:28.756101 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.756060 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5c84\" (UniqueName: \"kubernetes.io/projected/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-kube-api-access-f5c84\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb\" (UID: \"95bc0b39-5736-442b-96ab-d9e6ea6d63b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb" Nov 03 15:45:28.756221 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.756205 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb\" (UID: \"95bc0b39-5736-442b-96ab-d9e6ea6d63b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb" Nov 03 15:45:28.857004 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.856973 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb\" (UID: \"95bc0b39-5736-442b-96ab-d9e6ea6d63b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb" Nov 03 15:45:28.857174 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.857067 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5c84\" (UniqueName: \"kubernetes.io/projected/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-kube-api-access-f5c84\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb\" (UID: \"95bc0b39-5736-442b-96ab-d9e6ea6d63b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb" Nov 03 15:45:28.857255 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.857216 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb\" (UID: \"95bc0b39-5736-442b-96ab-d9e6ea6d63b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb" Nov 03 15:45:28.857670 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.857649 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb\" (UID: \"95bc0b39-5736-442b-96ab-d9e6ea6d63b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb" Nov 03 15:45:28.859297 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.859270 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb\" (UID: \"95bc0b39-5736-442b-96ab-d9e6ea6d63b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb" Nov 03 15:45:28.919487 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.919425 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5c84\" (UniqueName: \"kubernetes.io/projected/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-kube-api-access-f5c84\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb\" (UID: \"95bc0b39-5736-442b-96ab-d9e6ea6d63b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb" Nov 03 15:45:28.931986 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:28.931962 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb" Nov 03 15:45:29.311856 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:29.311811 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb"] Nov 03 15:45:29.313650 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:45:29.313625 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95bc0b39_5736_442b_96ab_d9e6ea6d63b0.slice/crio-ebf43e08a91c6f99e4e74cddffd00c1e9d6a8a5590890021f9fefb480c4a9105 WatchSource:0}: Error finding container ebf43e08a91c6f99e4e74cddffd00c1e9d6a8a5590890021f9fefb480c4a9105: Status 404 returned error can't find the container with id ebf43e08a91c6f99e4e74cddffd00c1e9d6a8a5590890021f9fefb480c4a9105 Nov 03 15:45:30.164698 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:30.164671 4557 generic.go:356] "Generic (PLEG): container finished" podID="95bc0b39-5736-442b-96ab-d9e6ea6d63b0" containerID="75446c073bde33da85165e6f6c0f560a3d0ad5eeaf20e3065acf5a039c3b4cc2" exitCode=0 Nov 03 15:45:30.165105 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:30.164711 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb" event={"ID":"95bc0b39-5736-442b-96ab-d9e6ea6d63b0","Type":"ContainerDied","Data":"75446c073bde33da85165e6f6c0f560a3d0ad5eeaf20e3065acf5a039c3b4cc2"} Nov 03 15:45:30.165105 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:30.164749 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb" event={"ID":"95bc0b39-5736-442b-96ab-d9e6ea6d63b0","Type":"ContainerStarted","Data":"ebf43e08a91c6f99e4e74cddffd00c1e9d6a8a5590890021f9fefb480c4a9105"} Nov 03 15:45:30.880123 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:30.880092 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:31.078324 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:31.078291 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:31.270191 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:31.270155 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:31.339126 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:31.339102 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb" Nov 03 15:45:31.489613 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:31.489516 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-kube-api-access-f5c84\" (UniqueName: \"kubernetes.io/projected/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-kube-api-access-f5c84\") pod \"95bc0b39-5736-442b-96ab-d9e6ea6d63b0\" (UID: \"95bc0b39-5736-442b-96ab-d9e6ea6d63b0\") " Nov 03 15:45:31.489783 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:31.489651 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-maas-api-openshift-ca\") pod \"95bc0b39-5736-442b-96ab-d9e6ea6d63b0\" (UID: \"95bc0b39-5736-442b-96ab-d9e6ea6d63b0\") " Nov 03 15:45:31.489783 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:31.489726 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-maas-api-xks-ca\") pod \"95bc0b39-5736-442b-96ab-d9e6ea6d63b0\" (UID: \"95bc0b39-5736-442b-96ab-d9e6ea6d63b0\") " Nov 03 15:45:31.490082 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:31.490053 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-maas-api-openshift-ca" pod "95bc0b39-5736-442b-96ab-d9e6ea6d63b0" (UID: "95bc0b39-5736-442b-96ab-d9e6ea6d63b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:31.490299 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:31.490281 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:31.492230 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:31.492195 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-kube-api-access-f5c84" pod "95bc0b39-5736-442b-96ab-d9e6ea6d63b0" (UID: "95bc0b39-5736-442b-96ab-d9e6ea6d63b0"). InnerVolumeSpecName "kube-api-access-f5c84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:31.492341 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:31.492282 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-maas-api-xks-ca" pod "95bc0b39-5736-442b-96ab-d9e6ea6d63b0" (UID: "95bc0b39-5736-442b-96ab-d9e6ea6d63b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:45:31.591621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:31.591565 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:31.591621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:31.591619 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5c84\" (UniqueName: \"kubernetes.io/projected/95bc0b39-5736-442b-96ab-d9e6ea6d63b0-kube-api-access-f5c84\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:32.194470 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:32.194436 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb" event={"ID":"95bc0b39-5736-442b-96ab-d9e6ea6d63b0","Type":"ContainerDied","Data":"ebf43e08a91c6f99e4e74cddffd00c1e9d6a8a5590890021f9fefb480c4a9105"} Nov 03 15:45:32.194716 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:32.194494 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebf43e08a91c6f99e4e74cddffd00c1e9d6a8a5590890021f9fefb480c4a9105" Nov 03 15:45:32.194716 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:32.194564 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb" Nov 03 15:45:35.581336 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:35.581294 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:36.778851 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:36.778816 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:37.150277 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:37.150245 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:37.473075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:37.472982 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:38.137863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:38.137816 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:38.417507 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:38.417398 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:38.728366 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:38.728264 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:38.929215 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:38.929184 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:39.479967 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:39.479933 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:40.147343 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:40.147310 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8"] Nov 03 15:45:40.261033 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:40.260995 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l"] Nov 03 15:45:40.549326 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:40.549300 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:40.738531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:40.738499 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:44.142223 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:44.142140 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:44.980030 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:44.979996 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:45.147664 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.147627 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8" podUID="dabfda2f-cfec-47ec-a322-d5bf5eaceba6" containerName="payload-processing" containerID="cri-o://c99c09c6acb1deaab80fc0e1c837db0d583ee1801d96c87add054f00be48bdbd" gracePeriod=25 Nov 03 15:45:45.193621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.193594 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:45.261600 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.261502 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l" podUID="8028cf00-9a15-49a8-8551-5391baf17422" containerName="payload-pre-processing" containerID="cri-o://03c0dc329fcdb8b8d1f1255d273b71bfc15f1cd939abdf1948b4f95cd64ae756" gracePeriod=25 Nov 03 15:45:45.283079 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.283049 4557 generic.go:356] "Generic (PLEG): container finished" podID="dabfda2f-cfec-47ec-a322-d5bf5eaceba6" containerID="c99c09c6acb1deaab80fc0e1c837db0d583ee1801d96c87add054f00be48bdbd" exitCode=0 Nov 03 15:45:45.283417 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.283397 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8" event={"ID":"dabfda2f-cfec-47ec-a322-d5bf5eaceba6","Type":"ContainerDied","Data":"c99c09c6acb1deaab80fc0e1c837db0d583ee1801d96c87add054f00be48bdbd"} Nov 03 15:45:45.423610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.420450 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:45.520659 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.519221 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8" Nov 03 15:45:45.604338 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.604311 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l" Nov 03 15:45:45.692383 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.692355 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dabfda2f-cfec-47ec-a322-d5bf5eaceba6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dabfda2f-cfec-47ec-a322-d5bf5eaceba6-plugins-config-volume\") pod \"dabfda2f-cfec-47ec-a322-d5bf5eaceba6\" (UID: \"dabfda2f-cfec-47ec-a322-d5bf5eaceba6\") " Nov 03 15:45:45.692553 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.692445 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dabfda2f-cfec-47ec-a322-d5bf5eaceba6-kube-api-access-6xqr6\" (UniqueName: \"kubernetes.io/projected/dabfda2f-cfec-47ec-a322-d5bf5eaceba6-kube-api-access-6xqr6\") pod \"dabfda2f-cfec-47ec-a322-d5bf5eaceba6\" (UID: \"dabfda2f-cfec-47ec-a322-d5bf5eaceba6\") " Nov 03 15:45:45.692764 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.692742 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dabfda2f-cfec-47ec-a322-d5bf5eaceba6-plugins-config-volume" pod "dabfda2f-cfec-47ec-a322-d5bf5eaceba6" (UID: "dabfda2f-cfec-47ec-a322-d5bf5eaceba6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:45.692908 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.692895 4557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dabfda2f-cfec-47ec-a322-d5bf5eaceba6-plugins-config-volume\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:45.694449 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.694430 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dabfda2f-cfec-47ec-a322-d5bf5eaceba6-kube-api-access-6xqr6" pod "dabfda2f-cfec-47ec-a322-d5bf5eaceba6" (UID: "dabfda2f-cfec-47ec-a322-d5bf5eaceba6"). InnerVolumeSpecName "kube-api-access-6xqr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:45.794084 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.794057 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8028cf00-9a15-49a8-8551-5391baf17422-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8028cf00-9a15-49a8-8551-5391baf17422-plugins-config-volume\") pod \"8028cf00-9a15-49a8-8551-5391baf17422\" (UID: \"8028cf00-9a15-49a8-8551-5391baf17422\") " Nov 03 15:45:45.794320 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.794302 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8028cf00-9a15-49a8-8551-5391baf17422-kube-api-access-snvtx\" (UniqueName: \"kubernetes.io/projected/8028cf00-9a15-49a8-8551-5391baf17422-kube-api-access-snvtx\") pod \"8028cf00-9a15-49a8-8551-5391baf17422\" (UID: \"8028cf00-9a15-49a8-8551-5391baf17422\") " Nov 03 15:45:45.794488 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.794458 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8028cf00-9a15-49a8-8551-5391baf17422-plugins-config-volume" pod "8028cf00-9a15-49a8-8551-5391baf17422" (UID: "8028cf00-9a15-49a8-8551-5391baf17422"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:45.794750 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.794733 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xqr6\" (UniqueName: \"kubernetes.io/projected/dabfda2f-cfec-47ec-a322-d5bf5eaceba6-kube-api-access-6xqr6\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:45.794842 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.794752 4557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8028cf00-9a15-49a8-8551-5391baf17422-plugins-config-volume\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:45.796575 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.796547 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8028cf00-9a15-49a8-8551-5391baf17422-kube-api-access-snvtx" pod "8028cf00-9a15-49a8-8551-5391baf17422" (UID: "8028cf00-9a15-49a8-8551-5391baf17422"). InnerVolumeSpecName "kube-api-access-snvtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:45.815158 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.815137 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:45.895799 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:45.895775 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snvtx\" (UniqueName: \"kubernetes.io/projected/8028cf00-9a15-49a8-8551-5391baf17422-kube-api-access-snvtx\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:46.291388 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.291357 4557 generic.go:356] "Generic (PLEG): container finished" podID="8028cf00-9a15-49a8-8551-5391baf17422" containerID="03c0dc329fcdb8b8d1f1255d273b71bfc15f1cd939abdf1948b4f95cd64ae756" exitCode=0 Nov 03 15:45:46.291913 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.291542 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l" event={"ID":"8028cf00-9a15-49a8-8551-5391baf17422","Type":"ContainerDied","Data":"03c0dc329fcdb8b8d1f1255d273b71bfc15f1cd939abdf1948b4f95cd64ae756"} Nov 03 15:45:46.291913 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.291596 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l" Nov 03 15:45:46.291913 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.291624 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l" event={"ID":"8028cf00-9a15-49a8-8551-5391baf17422","Type":"ContainerDied","Data":"f712f51f6f04fb99249561643e1ee8bfbaea1f59a63a9426b275410d5d5c1421"} Nov 03 15:45:46.291913 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.291660 4557 scope.go:122] "RemoveContainer" containerID="03c0dc329fcdb8b8d1f1255d273b71bfc15f1cd939abdf1948b4f95cd64ae756" Nov 03 15:45:46.293901 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.293874 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8" event={"ID":"dabfda2f-cfec-47ec-a322-d5bf5eaceba6","Type":"ContainerDied","Data":"284f4809923b1f27d12866bf45d4ee6a2933a8a4350263070fe96cc7c74fb947"} Nov 03 15:45:46.294027 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.293912 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8" Nov 03 15:45:46.302572 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.302557 4557 scope.go:122] "RemoveContainer" containerID="03c0dc329fcdb8b8d1f1255d273b71bfc15f1cd939abdf1948b4f95cd64ae756" Nov 03 15:45:46.302860 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:46.302833 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c0dc329fcdb8b8d1f1255d273b71bfc15f1cd939abdf1948b4f95cd64ae756\": container with ID starting with 03c0dc329fcdb8b8d1f1255d273b71bfc15f1cd939abdf1948b4f95cd64ae756 not found: ID does not exist" containerID="03c0dc329fcdb8b8d1f1255d273b71bfc15f1cd939abdf1948b4f95cd64ae756" Nov 03 15:45:46.302944 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.302869 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c0dc329fcdb8b8d1f1255d273b71bfc15f1cd939abdf1948b4f95cd64ae756"} err="failed to get container status \"03c0dc329fcdb8b8d1f1255d273b71bfc15f1cd939abdf1948b4f95cd64ae756\": rpc error: code = NotFound desc = could not find container \"03c0dc329fcdb8b8d1f1255d273b71bfc15f1cd939abdf1948b4f95cd64ae756\": container with ID starting with 03c0dc329fcdb8b8d1f1255d273b71bfc15f1cd939abdf1948b4f95cd64ae756 not found: ID does not exist" Nov 03 15:45:46.302944 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.302895 4557 scope.go:122] "RemoveContainer" containerID="c99c09c6acb1deaab80fc0e1c837db0d583ee1801d96c87add054f00be48bdbd" Nov 03 15:45:46.405192 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.405158 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq"] Nov 03 15:45:46.405695 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.405676 4557 cpu_manager.go:414] "containerMap: removing container" podUID="8028cf00-9a15-49a8-8551-5391baf17422" containerName="payload-pre-processing" Nov 03 15:45:46.405810 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.405696 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8028cf00-9a15-49a8-8551-5391baf17422" containerName="payload-pre-processing" Nov 03 15:45:46.405810 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.405714 4557 cpu_manager.go:414] "containerMap: removing container" podUID="148ee23d-c6d4-42b6-9173-e16173fe1063" containerName="cleanup" Nov 03 15:45:46.405810 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.405722 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="148ee23d-c6d4-42b6-9173-e16173fe1063" containerName="cleanup" Nov 03 15:45:46.405810 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.405763 4557 cpu_manager.go:414] "containerMap: removing container" podUID="43aaa640-044f-43cd-ab66-089ed62e48fb" containerName="cleanup" Nov 03 15:45:46.405810 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.405771 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43aaa640-044f-43cd-ab66-089ed62e48fb" containerName="cleanup" Nov 03 15:45:46.405810 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.405795 4557 cpu_manager.go:414] "containerMap: removing container" podUID="c6f53996-4386-4d7f-be27-4730e4804945" containerName="cleanup" Nov 03 15:45:46.405810 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.405802 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f53996-4386-4d7f-be27-4730e4804945" containerName="cleanup" Nov 03 15:45:46.406146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.405816 4557 cpu_manager.go:414] "containerMap: removing container" podUID="dabfda2f-cfec-47ec-a322-d5bf5eaceba6" containerName="payload-processing" Nov 03 15:45:46.406146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.405823 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dabfda2f-cfec-47ec-a322-d5bf5eaceba6" containerName="payload-processing" Nov 03 15:45:46.406146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.405840 4557 cpu_manager.go:414] "containerMap: removing container" podUID="82aedd62-09e8-4bf9-aa75-fab98c1cf785" containerName="cleanup" Nov 03 15:45:46.406146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.405848 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82aedd62-09e8-4bf9-aa75-fab98c1cf785" containerName="cleanup" Nov 03 15:45:46.406146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.405858 4557 cpu_manager.go:414] "containerMap: removing container" podUID="95bc0b39-5736-442b-96ab-d9e6ea6d63b0" containerName="revoke-keys" Nov 03 15:45:46.406146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.405865 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95bc0b39-5736-442b-96ab-d9e6ea6d63b0" containerName="revoke-keys" Nov 03 15:45:46.406146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.405973 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="95bc0b39-5736-442b-96ab-d9e6ea6d63b0" containerName="revoke-keys" Nov 03 15:45:46.406146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.405990 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="8028cf00-9a15-49a8-8551-5391baf17422" containerName="payload-pre-processing" Nov 03 15:45:46.406146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.406003 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="6120a8ff-bed0-4512-bd06-09e7e917a26c" containerName="cleanup" Nov 03 15:45:46.406146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.406015 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="a177f815-6766-447d-afea-c04c1a5dc18d" containerName="cleanup" Nov 03 15:45:46.406146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.406024 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f53996-4386-4d7f-be27-4730e4804945" containerName="cleanup" Nov 03 15:45:46.406146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.406032 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="82aedd62-09e8-4bf9-aa75-fab98c1cf785" containerName="cleanup" Nov 03 15:45:46.406146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.406041 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="dabfda2f-cfec-47ec-a322-d5bf5eaceba6" containerName="payload-processing" Nov 03 15:45:46.410416 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.410397 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq" Nov 03 15:45:46.422690 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.422658 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq"] Nov 03 15:45:46.464136 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.464108 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:45:46.464296 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.464201 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:45:46.481177 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.481150 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8"] Nov 03 15:45:46.516941 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.516909 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f356bb-6c79f76899-bqch8"] Nov 03 15:45:46.530888 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:45:46.530858 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:45:46.601195 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.601163 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l"] Nov 03 15:45:46.602917 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.602885 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3395af-d9ae-443c-a43e-54aeacc17c81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq\" (UID: \"6a3395af-d9ae-443c-a43e-54aeacc17c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq" Nov 03 15:45:46.603047 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.603027 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3395af-d9ae-443c-a43e-54aeacc17c81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq\" (UID: \"6a3395af-d9ae-443c-a43e-54aeacc17c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq" Nov 03 15:45:46.603202 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.603170 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvmpn\" (UniqueName: \"kubernetes.io/projected/6a3395af-d9ae-443c-a43e-54aeacc17c81-kube-api-access-qvmpn\") pod \"maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq\" (UID: \"6a3395af-d9ae-443c-a43e-54aeacc17c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq" Nov 03 15:45:46.609119 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.609096 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f356bb-59f86664d7-4sz7l"] Nov 03 15:45:46.704896 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.704830 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmpn\" (UniqueName: \"kubernetes.io/projected/6a3395af-d9ae-443c-a43e-54aeacc17c81-kube-api-access-qvmpn\") pod \"maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq\" (UID: \"6a3395af-d9ae-443c-a43e-54aeacc17c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq" Nov 03 15:45:46.705009 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.704901 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3395af-d9ae-443c-a43e-54aeacc17c81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq\" (UID: \"6a3395af-d9ae-443c-a43e-54aeacc17c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq" Nov 03 15:45:46.705009 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.705001 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3395af-d9ae-443c-a43e-54aeacc17c81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq\" (UID: \"6a3395af-d9ae-443c-a43e-54aeacc17c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq" Nov 03 15:45:46.705534 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.705514 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3395af-d9ae-443c-a43e-54aeacc17c81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq\" (UID: \"6a3395af-d9ae-443c-a43e-54aeacc17c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq" Nov 03 15:45:46.707325 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.707303 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3395af-d9ae-443c-a43e-54aeacc17c81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq\" (UID: \"6a3395af-d9ae-443c-a43e-54aeacc17c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq" Nov 03 15:45:46.765949 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:46.765924 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvmpn\" (UniqueName: \"kubernetes.io/projected/6a3395af-d9ae-443c-a43e-54aeacc17c81-kube-api-access-qvmpn\") pod \"maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq\" (UID: \"6a3395af-d9ae-443c-a43e-54aeacc17c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq" Nov 03 15:45:47.016255 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:47.016186 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:47.024058 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:47.024038 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq" Nov 03 15:45:47.240169 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:47.240141 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:47.454605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:47.450238 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq"] Nov 03 15:45:47.498173 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:47.498122 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:48.303017 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:48.302982 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8028cf00-9a15-49a8-8551-5391baf17422" path="/var/lib/kubelet/pods/8028cf00-9a15-49a8-8551-5391baf17422/volumes" Nov 03 15:45:48.303566 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:48.303541 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dabfda2f-cfec-47ec-a322-d5bf5eaceba6" path="/var/lib/kubelet/pods/dabfda2f-cfec-47ec-a322-d5bf5eaceba6/volumes" Nov 03 15:45:48.311472 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:48.311438 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq" event={"ID":"6a3395af-d9ae-443c-a43e-54aeacc17c81","Type":"ContainerStarted","Data":"b042d841744b12df381f55f1347438ef8d61bf40b9f2d2231186bdba0b189354"} Nov 03 15:45:48.311612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:48.311496 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq" event={"ID":"6a3395af-d9ae-443c-a43e-54aeacc17c81","Type":"ContainerStarted","Data":"29f2bb1ccca1679886d22874f7c8f2c80d7550f2bf6bd870f4efdf56f7d50ecf"} Nov 03 15:45:48.438969 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:48.438917 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq" podStartSLOduration=2.438901995 podStartE2EDuration="2.438901995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:48.436507575 +0000 UTC m=+2258.279629122" watchObservedRunningTime="2026-11-03 15:45:48.438901995 +0000 UTC m=+2258.282023536" Nov 03 15:45:49.318782 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:49.318751 4557 generic.go:356] "Generic (PLEG): container finished" podID="6a3395af-d9ae-443c-a43e-54aeacc17c81" containerID="b042d841744b12df381f55f1347438ef8d61bf40b9f2d2231186bdba0b189354" exitCode=0 Nov 03 15:45:49.319169 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:49.318838 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq" event={"ID":"6a3395af-d9ae-443c-a43e-54aeacc17c81","Type":"ContainerDied","Data":"b042d841744b12df381f55f1347438ef8d61bf40b9f2d2231186bdba0b189354"} Nov 03 15:45:49.676118 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:49.676022 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:50.471760 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:50.471736 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq" Nov 03 15:45:50.574609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:50.574496 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3395af-d9ae-443c-a43e-54aeacc17c81-kube-api-access-qvmpn\" (UniqueName: \"kubernetes.io/projected/6a3395af-d9ae-443c-a43e-54aeacc17c81-kube-api-access-qvmpn\") pod \"6a3395af-d9ae-443c-a43e-54aeacc17c81\" (UID: \"6a3395af-d9ae-443c-a43e-54aeacc17c81\") " Nov 03 15:45:50.574780 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:50.574710 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a3395af-d9ae-443c-a43e-54aeacc17c81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3395af-d9ae-443c-a43e-54aeacc17c81-maas-api-openshift-ca\") pod \"6a3395af-d9ae-443c-a43e-54aeacc17c81\" (UID: \"6a3395af-d9ae-443c-a43e-54aeacc17c81\") " Nov 03 15:45:50.574780 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:50.574745 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a3395af-d9ae-443c-a43e-54aeacc17c81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3395af-d9ae-443c-a43e-54aeacc17c81-maas-api-xks-ca\") pod \"6a3395af-d9ae-443c-a43e-54aeacc17c81\" (UID: \"6a3395af-d9ae-443c-a43e-54aeacc17c81\") " Nov 03 15:45:50.575061 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:50.575032 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3395af-d9ae-443c-a43e-54aeacc17c81-maas-api-openshift-ca" pod "6a3395af-d9ae-443c-a43e-54aeacc17c81" (UID: "6a3395af-d9ae-443c-a43e-54aeacc17c81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:45:50.575193 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:50.575149 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3395af-d9ae-443c-a43e-54aeacc17c81-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:50.576690 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:50.576661 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3395af-d9ae-443c-a43e-54aeacc17c81-kube-api-access-qvmpn" pod "6a3395af-d9ae-443c-a43e-54aeacc17c81" (UID: "6a3395af-d9ae-443c-a43e-54aeacc17c81"). InnerVolumeSpecName "kube-api-access-qvmpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:45:50.576775 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:50.576740 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a3395af-d9ae-443c-a43e-54aeacc17c81-maas-api-xks-ca" pod "6a3395af-d9ae-443c-a43e-54aeacc17c81" (UID: "6a3395af-d9ae-443c-a43e-54aeacc17c81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:45:50.676536 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:50.676507 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3395af-d9ae-443c-a43e-54aeacc17c81-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:50.676536 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:50.676534 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvmpn\" (UniqueName: \"kubernetes.io/projected/6a3395af-d9ae-443c-a43e-54aeacc17c81-kube-api-access-qvmpn\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:45:50.769111 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:50.769072 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz"] Nov 03 15:45:50.985229 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:50.985151 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:51.331074 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:51.331040 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq" Nov 03 15:45:51.331237 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:51.331046 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq" event={"ID":"6a3395af-d9ae-443c-a43e-54aeacc17c81","Type":"ContainerDied","Data":"29f2bb1ccca1679886d22874f7c8f2c80d7550f2bf6bd870f4efdf56f7d50ecf"} Nov 03 15:45:51.331237 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:51.331161 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29f2bb1ccca1679886d22874f7c8f2c80d7550f2bf6bd870f4efdf56f7d50ecf" Nov 03 15:45:54.436704 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:54.436668 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:56.076929 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:56.076883 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:56.326540 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:56.326498 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:56.551054 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:56.551015 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:57.025524 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:57.025433 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:58.550208 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:58.550080 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:58.757293 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:58.757251 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:59.647605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:59.644759 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:45:59.874690 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:45:59.874653 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:01.025469 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:01.025433 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:01.265173 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:01.265142 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:01.720161 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:01.720074 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:02.076045 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:02.076012 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:05.789037 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:05.788979 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" containerID="cri-o://2e8e6c4272a708744a6f3a5768852a1b5f3a4a028cf885fe15fffd8d6daa2ae6" gracePeriod=45 Nov 03 15:46:05.906866 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:05.906838 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl"] Nov 03 15:46:05.907081 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:05.907055 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" podUID="76677f41-4a57-4d11-afb4-6fc9ec4c8299" containerName="maas-api" containerID="cri-o://86259576fe788fded66fd63e19b17a38a78e5167da652d83fa1e5764092fefcb" gracePeriod=30 Nov 03 15:46:06.043899 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.043861 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:06.080843 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.080812 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:46:06.193650 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.183072 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-model-cache\") pod \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " Nov 03 15:46:06.193650 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.183159 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-dshm\") pod \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " Nov 03 15:46:06.193650 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.183225 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-tls-certs\" (UniqueName: \"kubernetes.io/secret/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-tls-certs\") pod \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " Nov 03 15:46:06.193650 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.183264 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-kube-api-access-48nzz\" (UniqueName: \"kubernetes.io/projected/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-kube-api-access-48nzz\") pod \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " Nov 03 15:46:06.193650 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.183378 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-tmp-dir\") pod \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " Nov 03 15:46:06.193650 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.183406 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-home\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-home\") pod \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\" (UID: \"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1\") " Nov 03 15:46:06.194259 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.194209 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-home" pod "236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" (UID: "236d85e1-9c70-4f9a-8c6c-e41901f7bdc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:06.194788 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.194764 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-model-cache" pod "236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" (UID: "236d85e1-9c70-4f9a-8c6c-e41901f7bdc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:06.196284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.196204 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-tmp-dir" pod "236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" (UID: "236d85e1-9c70-4f9a-8c6c-e41901f7bdc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:06.198127 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.198086 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-tls-certs" pod "236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" (UID: "236d85e1-9c70-4f9a-8c6c-e41901f7bdc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:06.198750 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.198714 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-dshm" pod "236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" (UID: "236d85e1-9c70-4f9a-8c6c-e41901f7bdc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:06.204156 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.204110 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-kube-api-access-48nzz" pod "236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" (UID: "236d85e1-9c70-4f9a-8c6c-e41901f7bdc1"). InnerVolumeSpecName "kube-api-access-48nzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:06.285266 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.285232 4557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-model-cache\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:06.285266 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.285263 4557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-dshm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:06.285266 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.285272 4557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-tls-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:06.285266 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.285281 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48nzz\" (UniqueName: \"kubernetes.io/projected/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-kube-api-access-48nzz\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:06.285676 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.285290 4557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-tmp-dir\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:06.285676 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.285298 4557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1-home\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:06.319645 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.319540 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:46:06.385824 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.385794 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76677f41-4a57-4d11-afb4-6fc9ec4c8299-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76677f41-4a57-4d11-afb4-6fc9ec4c8299-maas-api-tls\") pod \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\" (UID: \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\") " Nov 03 15:46:06.385991 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.385859 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76677f41-4a57-4d11-afb4-6fc9ec4c8299-kube-api-access-vkcmg\" (UniqueName: \"kubernetes.io/projected/76677f41-4a57-4d11-afb4-6fc9ec4c8299-kube-api-access-vkcmg\") pod \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\" (UID: \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\") " Nov 03 15:46:06.385991 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.385923 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76677f41-4a57-4d11-afb4-6fc9ec4c8299-metrics-certs\" (UniqueName: \"kubernetes.io/secret/76677f41-4a57-4d11-afb4-6fc9ec4c8299-metrics-certs\") pod \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\" (UID: \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\") " Nov 03 15:46:06.386092 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.386075 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76677f41-4a57-4d11-afb4-6fc9ec4c8299-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76677f41-4a57-4d11-afb4-6fc9ec4c8299-openshift-service-ca\") pod \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\" (UID: \"76677f41-4a57-4d11-afb4-6fc9ec4c8299\") " Nov 03 15:46:06.386417 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.386393 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76677f41-4a57-4d11-afb4-6fc9ec4c8299-openshift-service-ca" pod "76677f41-4a57-4d11-afb4-6fc9ec4c8299" (UID: "76677f41-4a57-4d11-afb4-6fc9ec4c8299"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:06.386748 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.386723 4557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76677f41-4a57-4d11-afb4-6fc9ec4c8299-openshift-service-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:06.388046 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.388020 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76677f41-4a57-4d11-afb4-6fc9ec4c8299-metrics-certs" pod "76677f41-4a57-4d11-afb4-6fc9ec4c8299" (UID: "76677f41-4a57-4d11-afb4-6fc9ec4c8299"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:06.388148 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.388025 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76677f41-4a57-4d11-afb4-6fc9ec4c8299-kube-api-access-vkcmg" pod "76677f41-4a57-4d11-afb4-6fc9ec4c8299" (UID: "76677f41-4a57-4d11-afb4-6fc9ec4c8299"). InnerVolumeSpecName "kube-api-access-vkcmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:06.388233 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.388215 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76677f41-4a57-4d11-afb4-6fc9ec4c8299-maas-api-tls" pod "76677f41-4a57-4d11-afb4-6fc9ec4c8299" (UID: "76677f41-4a57-4d11-afb4-6fc9ec4c8299"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:06.491604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.488312 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76677f41-4a57-4d11-afb4-6fc9ec4c8299-maas-api-tls\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:06.491604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.488348 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkcmg\" (UniqueName: \"kubernetes.io/projected/76677f41-4a57-4d11-afb4-6fc9ec4c8299-kube-api-access-vkcmg\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:06.491604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.488363 4557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76677f41-4a57-4d11-afb4-6fc9ec4c8299-metrics-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:06.496621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.493699 4557 generic.go:356] "Generic (PLEG): container finished" podID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerID="2e8e6c4272a708744a6f3a5768852a1b5f3a4a028cf885fe15fffd8d6daa2ae6" exitCode=0 Nov 03 15:46:06.496621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.493838 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" event={"ID":"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1","Type":"ContainerDied","Data":"2e8e6c4272a708744a6f3a5768852a1b5f3a4a028cf885fe15fffd8d6daa2ae6"} Nov 03 15:46:06.496621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.493882 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" event={"ID":"236d85e1-9c70-4f9a-8c6c-e41901f7bdc1","Type":"ContainerDied","Data":"edcf99eae7743d5011694d637312a74c6226bc20dc509ee6ad63e6abd14b42e7"} Nov 03 15:46:06.496621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.493905 4557 scope.go:122] "RemoveContainer" containerID="2e8e6c4272a708744a6f3a5768852a1b5f3a4a028cf885fe15fffd8d6daa2ae6" Nov 03 15:46:06.496621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.494060 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" Nov 03 15:46:06.501527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.501364 4557 generic.go:356] "Generic (PLEG): container finished" podID="76677f41-4a57-4d11-afb4-6fc9ec4c8299" containerID="86259576fe788fded66fd63e19b17a38a78e5167da652d83fa1e5764092fefcb" exitCode=0 Nov 03 15:46:06.501527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.501433 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" Nov 03 15:46:06.501527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.501442 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" event={"ID":"76677f41-4a57-4d11-afb4-6fc9ec4c8299","Type":"ContainerDied","Data":"86259576fe788fded66fd63e19b17a38a78e5167da652d83fa1e5764092fefcb"} Nov 03 15:46:06.501527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.501502 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" event={"ID":"76677f41-4a57-4d11-afb4-6fc9ec4c8299","Type":"ContainerDied","Data":"af4faf7594c490ca3b809e5aad8c0a5ef1dcf783620af0319c9fa1a090383fd9"} Nov 03 15:46:06.510655 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.510631 4557 scope.go:122] "RemoveContainer" containerID="2e8e6c4272a708744a6f3a5768852a1b5f3a4a028cf885fe15fffd8d6daa2ae6" Nov 03 15:46:06.510969 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:46:06.510948 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e8e6c4272a708744a6f3a5768852a1b5f3a4a028cf885fe15fffd8d6daa2ae6\": container with ID starting with 2e8e6c4272a708744a6f3a5768852a1b5f3a4a028cf885fe15fffd8d6daa2ae6 not found: ID does not exist" containerID="2e8e6c4272a708744a6f3a5768852a1b5f3a4a028cf885fe15fffd8d6daa2ae6" Nov 03 15:46:06.511062 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.510977 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e8e6c4272a708744a6f3a5768852a1b5f3a4a028cf885fe15fffd8d6daa2ae6"} err="failed to get container status \"2e8e6c4272a708744a6f3a5768852a1b5f3a4a028cf885fe15fffd8d6daa2ae6\": rpc error: code = NotFound desc = could not find container \"2e8e6c4272a708744a6f3a5768852a1b5f3a4a028cf885fe15fffd8d6daa2ae6\": container with ID starting with 2e8e6c4272a708744a6f3a5768852a1b5f3a4a028cf885fe15fffd8d6daa2ae6 not found: ID does not exist" Nov 03 15:46:06.511062 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.510996 4557 scope.go:122] "RemoveContainer" containerID="86259576fe788fded66fd63e19b17a38a78e5167da652d83fa1e5764092fefcb" Nov 03 15:46:06.519767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.519745 4557 scope.go:122] "RemoveContainer" containerID="86259576fe788fded66fd63e19b17a38a78e5167da652d83fa1e5764092fefcb" Nov 03 15:46:06.520039 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:46:06.520018 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86259576fe788fded66fd63e19b17a38a78e5167da652d83fa1e5764092fefcb\": container with ID starting with 86259576fe788fded66fd63e19b17a38a78e5167da652d83fa1e5764092fefcb not found: ID does not exist" containerID="86259576fe788fded66fd63e19b17a38a78e5167da652d83fa1e5764092fefcb" Nov 03 15:46:06.520117 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.520068 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86259576fe788fded66fd63e19b17a38a78e5167da652d83fa1e5764092fefcb"} err="failed to get container status \"86259576fe788fded66fd63e19b17a38a78e5167da652d83fa1e5764092fefcb\": rpc error: code = NotFound desc = could not find container \"86259576fe788fded66fd63e19b17a38a78e5167da652d83fa1e5764092fefcb\": container with ID starting with 86259576fe788fded66fd63e19b17a38a78e5167da652d83fa1e5764092fefcb not found: ID does not exist" Nov 03 15:46:06.665074 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.664927 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz"] Nov 03 15:46:06.674125 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.674084 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz"] Nov 03 15:46:06.783095 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.783063 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl"] Nov 03 15:46:06.789226 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.789197 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl"] Nov 03 15:46:06.981809 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:06.981718 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/simulator-w1-9855ff-kserve-58df8cfcdf-wlrxz" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 03 15:46:07.674140 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:07.674107 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:07.912791 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:07.912740 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:08.112337 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:08.112303 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:08.175656 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:08.175622 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb"] Nov 03 15:46:08.179635 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:08.179605 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f356bb-ssqjb"] Nov 03 15:46:08.284757 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:08.284727 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" path="/var/lib/kubelet/pods/236d85e1-9c70-4f9a-8c6c-e41901f7bdc1/volumes" Nov 03 15:46:08.285206 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:08.285192 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76677f41-4a57-4d11-afb4-6fc9ec4c8299" path="/var/lib/kubelet/pods/76677f41-4a57-4d11-afb4-6fc9ec4c8299/volumes" Nov 03 15:46:08.285625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:08.285611 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95bc0b39-5736-442b-96ab-d9e6ea6d63b0" path="/var/lib/kubelet/pods/95bc0b39-5736-442b-96ab-d9e6ea6d63b0/volumes" Nov 03 15:46:08.694703 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:08.694664 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:08.892284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:08.892252 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:09.219514 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:09.219468 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f356bb-59c7469d6c-2zzdl" podUID="76677f41-4a57-4d11-afb4-6fc9ec4c8299" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.149:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:46:13.463495 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:13.463429 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:15.332986 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:15.331970 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:15.552936 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:15.552903 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:15.821605 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:15.802029 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:16.234031 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.233939 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:16.530094 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:46:16.530010 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:46:16.771550 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.771519 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg"] Nov 03 15:46:16.772001 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.771986 4557 cpu_manager.go:414] "containerMap: removing container" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" Nov 03 15:46:16.772052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.772001 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" Nov 03 15:46:16.772052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.772023 4557 cpu_manager.go:414] "containerMap: removing container" podUID="6a3395af-d9ae-443c-a43e-54aeacc17c81" containerName="revoke-keys" Nov 03 15:46:16.772052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.772028 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3395af-d9ae-443c-a43e-54aeacc17c81" containerName="revoke-keys" Nov 03 15:46:16.772052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.772048 4557 cpu_manager.go:414] "containerMap: removing container" podUID="76677f41-4a57-4d11-afb4-6fc9ec4c8299" containerName="maas-api" Nov 03 15:46:16.772052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.772053 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76677f41-4a57-4d11-afb4-6fc9ec4c8299" containerName="maas-api" Nov 03 15:46:16.772210 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.772116 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a3395af-d9ae-443c-a43e-54aeacc17c81" containerName="revoke-keys" Nov 03 15:46:16.772210 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.772127 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="76677f41-4a57-4d11-afb4-6fc9ec4c8299" containerName="maas-api" Nov 03 15:46:16.772210 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.772136 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="236d85e1-9c70-4f9a-8c6c-e41901f7bdc1" containerName="main" Nov 03 15:46:16.776619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.776602 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg" Nov 03 15:46:16.826787 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.826759 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6428b6c6-80e9-485a-9884-f5933b41d5a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg\" (UID: \"6428b6c6-80e9-485a-9884-f5933b41d5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg" Nov 03 15:46:16.826921 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.826793 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm8x7\" (UniqueName: \"kubernetes.io/projected/6428b6c6-80e9-485a-9884-f5933b41d5a8-kube-api-access-pm8x7\") pod \"maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg\" (UID: \"6428b6c6-80e9-485a-9884-f5933b41d5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg" Nov 03 15:46:16.826921 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.826819 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6428b6c6-80e9-485a-9884-f5933b41d5a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg\" (UID: \"6428b6c6-80e9-485a-9884-f5933b41d5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg" Nov 03 15:46:16.833445 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.833423 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:46:16.836478 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.836443 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg"] Nov 03 15:46:16.883055 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.883032 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:46:16.928356 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.928334 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6428b6c6-80e9-485a-9884-f5933b41d5a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg\" (UID: \"6428b6c6-80e9-485a-9884-f5933b41d5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg" Nov 03 15:46:16.928455 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.928375 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm8x7\" (UniqueName: \"kubernetes.io/projected/6428b6c6-80e9-485a-9884-f5933b41d5a8-kube-api-access-pm8x7\") pod \"maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg\" (UID: \"6428b6c6-80e9-485a-9884-f5933b41d5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg" Nov 03 15:46:16.928455 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.928400 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6428b6c6-80e9-485a-9884-f5933b41d5a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg\" (UID: \"6428b6c6-80e9-485a-9884-f5933b41d5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg" Nov 03 15:46:16.929102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.929080 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6428b6c6-80e9-485a-9884-f5933b41d5a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg\" (UID: \"6428b6c6-80e9-485a-9884-f5933b41d5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg" Nov 03 15:46:16.931312 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.931288 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6428b6c6-80e9-485a-9884-f5933b41d5a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg\" (UID: \"6428b6c6-80e9-485a-9884-f5933b41d5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg" Nov 03 15:46:16.990054 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:16.990028 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm8x7\" (UniqueName: \"kubernetes.io/projected/6428b6c6-80e9-485a-9884-f5933b41d5a8-kube-api-access-pm8x7\") pod \"maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg\" (UID: \"6428b6c6-80e9-485a-9884-f5933b41d5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg" Nov 03 15:46:17.089253 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:17.089228 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg" Nov 03 15:46:17.470209 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:17.470175 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg"] Nov 03 15:46:17.471137 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:46:17.471112 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6428b6c6_80e9_485a_9884_f5933b41d5a8.slice/crio-b5f29a9fb01ef58e008ae93579fe8bed3411dfea2331058d4f780f8eac959f44 WatchSource:0}: Error finding container b5f29a9fb01ef58e008ae93579fe8bed3411dfea2331058d4f780f8eac959f44: Status 404 returned error can't find the container with id b5f29a9fb01ef58e008ae93579fe8bed3411dfea2331058d4f780f8eac959f44 Nov 03 15:46:17.566935 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:17.566900 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg" event={"ID":"6428b6c6-80e9-485a-9884-f5933b41d5a8","Type":"ContainerStarted","Data":"b5f29a9fb01ef58e008ae93579fe8bed3411dfea2331058d4f780f8eac959f44"} Nov 03 15:46:18.190473 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:18.190427 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:18.383890 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:18.383859 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:18.574247 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:18.574204 4557 generic.go:356] "Generic (PLEG): container finished" podID="6428b6c6-80e9-485a-9884-f5933b41d5a8" containerID="3c11b3d3b836e0657be7ef2b72ca4db0d28b222e06a3d1c8eea6939aa76583e3" exitCode=0 Nov 03 15:46:18.574674 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:18.574253 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg" event={"ID":"6428b6c6-80e9-485a-9884-f5933b41d5a8","Type":"ContainerDied","Data":"3c11b3d3b836e0657be7ef2b72ca4db0d28b222e06a3d1c8eea6939aa76583e3"} Nov 03 15:46:19.245485 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:19.245450 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:19.660263 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:19.660208 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:19.734640 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:19.734615 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg" Nov 03 15:46:19.862113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:19.862018 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6428b6c6-80e9-485a-9884-f5933b41d5a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6428b6c6-80e9-485a-9884-f5933b41d5a8-maas-api-openshift-ca\") pod \"6428b6c6-80e9-485a-9884-f5933b41d5a8\" (UID: \"6428b6c6-80e9-485a-9884-f5933b41d5a8\") " Nov 03 15:46:19.862113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:19.862101 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6428b6c6-80e9-485a-9884-f5933b41d5a8-kube-api-access-pm8x7\" (UniqueName: \"kubernetes.io/projected/6428b6c6-80e9-485a-9884-f5933b41d5a8-kube-api-access-pm8x7\") pod \"6428b6c6-80e9-485a-9884-f5933b41d5a8\" (UID: \"6428b6c6-80e9-485a-9884-f5933b41d5a8\") " Nov 03 15:46:19.862362 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:19.862249 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6428b6c6-80e9-485a-9884-f5933b41d5a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6428b6c6-80e9-485a-9884-f5933b41d5a8-maas-api-xks-ca\") pod \"6428b6c6-80e9-485a-9884-f5933b41d5a8\" (UID: \"6428b6c6-80e9-485a-9884-f5933b41d5a8\") " Nov 03 15:46:19.862489 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:19.862458 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6428b6c6-80e9-485a-9884-f5933b41d5a8-maas-api-openshift-ca" pod "6428b6c6-80e9-485a-9884-f5933b41d5a8" (UID: "6428b6c6-80e9-485a-9884-f5933b41d5a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:19.862841 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:19.862814 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6428b6c6-80e9-485a-9884-f5933b41d5a8-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:19.865103 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:19.865076 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6428b6c6-80e9-485a-9884-f5933b41d5a8-maas-api-xks-ca" pod "6428b6c6-80e9-485a-9884-f5933b41d5a8" (UID: "6428b6c6-80e9-485a-9884-f5933b41d5a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:19.865213 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:19.865189 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6428b6c6-80e9-485a-9884-f5933b41d5a8-kube-api-access-pm8x7" pod "6428b6c6-80e9-485a-9884-f5933b41d5a8" (UID: "6428b6c6-80e9-485a-9884-f5933b41d5a8"). InnerVolumeSpecName "kube-api-access-pm8x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:19.911066 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:19.911032 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:19.964012 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:19.963978 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm8x7\" (UniqueName: \"kubernetes.io/projected/6428b6c6-80e9-485a-9884-f5933b41d5a8-kube-api-access-pm8x7\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:19.964012 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:19.964007 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6428b6c6-80e9-485a-9884-f5933b41d5a8-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:20.588100 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:20.588018 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg" event={"ID":"6428b6c6-80e9-485a-9884-f5933b41d5a8","Type":"ContainerDied","Data":"b5f29a9fb01ef58e008ae93579fe8bed3411dfea2331058d4f780f8eac959f44"} Nov 03 15:46:20.588100 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:20.588068 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5f29a9fb01ef58e008ae93579fe8bed3411dfea2331058d4f780f8eac959f44" Nov 03 15:46:20.588100 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:20.588031 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg" Nov 03 15:46:21.170686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:21.170650 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45"] Nov 03 15:46:21.221279 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:21.221244 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv"] Nov 03 15:46:21.331625 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:21.331563 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:23.831988 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:23.831912 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:25.474536 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:25.474501 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:25.714878 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:25.714840 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:25.903705 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:25.903674 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:26.449364 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:26.449323 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:28.811286 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:28.811255 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:28.999888 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:28.999858 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:30.908457 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:30.908421 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:32.026994 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.026962 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:32.182892 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.182852 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt"] Nov 03 15:46:32.183389 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.183371 4557 cpu_manager.go:414] "containerMap: removing container" podUID="6428b6c6-80e9-485a-9884-f5933b41d5a8" containerName="revoke-keys" Nov 03 15:46:32.183487 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.183391 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6428b6c6-80e9-485a-9884-f5933b41d5a8" containerName="revoke-keys" Nov 03 15:46:32.183559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.183551 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="6428b6c6-80e9-485a-9884-f5933b41d5a8" containerName="revoke-keys" Nov 03 15:46:32.186811 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.186790 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt" Nov 03 15:46:32.303956 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.303927 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:46:32.304109 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.304018 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:46:32.306702 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.306665 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f42a5c8-cl4jt\" (UID: \"320d7c16-9625-45ed-bd30-56c0eaf2bbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt" Nov 03 15:46:32.306832 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.306723 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4r45\" (UniqueName: \"kubernetes.io/projected/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-kube-api-access-q4r45\") pod \"maas-api-revoke-sub-6f42a5c8-cl4jt\" (UID: \"320d7c16-9625-45ed-bd30-56c0eaf2bbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt" Nov 03 15:46:32.306878 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.306863 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt"] Nov 03 15:46:32.307032 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.307009 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f42a5c8-cl4jt\" (UID: \"320d7c16-9625-45ed-bd30-56c0eaf2bbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt" Nov 03 15:46:32.407695 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.407658 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f42a5c8-cl4jt\" (UID: \"320d7c16-9625-45ed-bd30-56c0eaf2bbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt" Nov 03 15:46:32.407863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.407704 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4r45\" (UniqueName: \"kubernetes.io/projected/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-kube-api-access-q4r45\") pod \"maas-api-revoke-sub-6f42a5c8-cl4jt\" (UID: \"320d7c16-9625-45ed-bd30-56c0eaf2bbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt" Nov 03 15:46:32.407863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.407850 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f42a5c8-cl4jt\" (UID: \"320d7c16-9625-45ed-bd30-56c0eaf2bbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt" Nov 03 15:46:32.408328 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.408299 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f42a5c8-cl4jt\" (UID: \"320d7c16-9625-45ed-bd30-56c0eaf2bbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt" Nov 03 15:46:32.410312 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.410289 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f42a5c8-cl4jt\" (UID: \"320d7c16-9625-45ed-bd30-56c0eaf2bbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt" Nov 03 15:46:32.426508 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.426483 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:32.480740 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.480716 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4r45\" (UniqueName: \"kubernetes.io/projected/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-kube-api-access-q4r45\") pod \"maas-api-revoke-sub-6f42a5c8-cl4jt\" (UID: \"320d7c16-9625-45ed-bd30-56c0eaf2bbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt" Nov 03 15:46:32.499546 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.499522 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt" Nov 03 15:46:32.743016 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:32.742984 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:33.001472 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:46:33.001433 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320d7c16_9625_45ed_bd30_56c0eaf2bbf9.slice/crio-fcf29ba67dbcac324d5be0484cfd668e26faf986ae6b21f422cd74444e3dec47 WatchSource:0}: Error finding container fcf29ba67dbcac324d5be0484cfd668e26faf986ae6b21f422cd74444e3dec47: Status 404 returned error can't find the container with id fcf29ba67dbcac324d5be0484cfd668e26faf986ae6b21f422cd74444e3dec47 Nov 03 15:46:33.074677 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:33.074640 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt"] Nov 03 15:46:33.134853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:33.134810 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:33.297425 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:33.297400 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq"] Nov 03 15:46:33.307526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:33.307503 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90fcca-54zhq"] Nov 03 15:46:33.483807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:33.483775 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:33.715406 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:33.715370 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt" event={"ID":"320d7c16-9625-45ed-bd30-56c0eaf2bbf9","Type":"ContainerStarted","Data":"b131eeba5fcab0e7b86e6788896486450f6b9c167b4f178d866ce3ea6318e975"} Nov 03 15:46:33.715628 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:33.715436 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt" event={"ID":"320d7c16-9625-45ed-bd30-56c0eaf2bbf9","Type":"ContainerStarted","Data":"fcf29ba67dbcac324d5be0484cfd668e26faf986ae6b21f422cd74444e3dec47"} Nov 03 15:46:33.897599 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:33.897532 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt" podStartSLOduration=1.897516143 podStartE2EDuration="1.897516143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:46:33.845421003 +0000 UTC m=+2303.688542555" watchObservedRunningTime="2026-11-03 15:46:33.897516143 +0000 UTC m=+2303.740637686" Nov 03 15:46:33.902249 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:33.902214 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:34.285029 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:34.284997 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce046a64-2b99-43e2-a354-a4129e45af72" path="/var/lib/kubelet/pods/ce046a64-2b99-43e2-a354-a4129e45af72/volumes" Nov 03 15:46:34.722830 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:34.722798 4557 generic.go:356] "Generic (PLEG): container finished" podID="320d7c16-9625-45ed-bd30-56c0eaf2bbf9" containerID="b131eeba5fcab0e7b86e6788896486450f6b9c167b4f178d866ce3ea6318e975" exitCode=0 Nov 03 15:46:34.723041 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:34.722872 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt" event={"ID":"320d7c16-9625-45ed-bd30-56c0eaf2bbf9","Type":"ContainerDied","Data":"b131eeba5fcab0e7b86e6788896486450f6b9c167b4f178d866ce3ea6318e975"} Nov 03 15:46:35.868571 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:35.868548 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt" Nov 03 15:46:35.952966 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:35.952926 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-kube-api-access-q4r45\" (UniqueName: \"kubernetes.io/projected/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-kube-api-access-q4r45\") pod \"320d7c16-9625-45ed-bd30-56c0eaf2bbf9\" (UID: \"320d7c16-9625-45ed-bd30-56c0eaf2bbf9\") " Nov 03 15:46:35.953160 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:35.953035 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-maas-api-openshift-ca\") pod \"320d7c16-9625-45ed-bd30-56c0eaf2bbf9\" (UID: \"320d7c16-9625-45ed-bd30-56c0eaf2bbf9\") " Nov 03 15:46:35.953160 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:35.953134 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-maas-api-xks-ca\") pod \"320d7c16-9625-45ed-bd30-56c0eaf2bbf9\" (UID: \"320d7c16-9625-45ed-bd30-56c0eaf2bbf9\") " Nov 03 15:46:35.953433 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:35.953407 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-maas-api-openshift-ca" pod "320d7c16-9625-45ed-bd30-56c0eaf2bbf9" (UID: "320d7c16-9625-45ed-bd30-56c0eaf2bbf9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:35.953691 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:35.953666 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:35.955082 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:35.955059 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-kube-api-access-q4r45" pod "320d7c16-9625-45ed-bd30-56c0eaf2bbf9" (UID: "320d7c16-9625-45ed-bd30-56c0eaf2bbf9"). InnerVolumeSpecName "kube-api-access-q4r45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:35.955155 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:35.955082 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-maas-api-xks-ca" pod "320d7c16-9625-45ed-bd30-56c0eaf2bbf9" (UID: "320d7c16-9625-45ed-bd30-56c0eaf2bbf9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:36.054640 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.054573 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.054640 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.054637 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4r45\" (UniqueName: \"kubernetes.io/projected/320d7c16-9625-45ed-bd30-56c0eaf2bbf9-kube-api-access-q4r45\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.191127 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.191068 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" containerID="cri-o://9641131ce77b6edaabf2ee148f3d4857eaf0a2e170668e9ab221df8baf2e7ebf" gracePeriod=45 Nov 03 15:46:36.243078 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.243035 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" containerID="cri-o://fb487b339f61a65fa6470bffa59815141496ae641973c69fbbc00e01a6e5507a" gracePeriod=45 Nov 03 15:46:36.503663 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.503633 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:46:36.527306 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.527281 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:46:36.559846 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.559815 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-tmp-dir\") pod \"8499b913-2f91-43f1-bede-5a7462494ef8\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " Nov 03 15:46:36.559964 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.559884 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-model-cache\") pod \"8499b913-2f91-43f1-bede-5a7462494ef8\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " Nov 03 15:46:36.560039 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.559968 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8499b913-2f91-43f1-bede-5a7462494ef8-kube-api-access-9mp4t\" (UniqueName: \"kubernetes.io/projected/8499b913-2f91-43f1-bede-5a7462494ef8-kube-api-access-9mp4t\") pod \"8499b913-2f91-43f1-bede-5a7462494ef8\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " Nov 03 15:46:36.560039 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.560010 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-home\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-home\") pod \"8499b913-2f91-43f1-bede-5a7462494ef8\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " Nov 03 15:46:36.560039 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.560023 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-tmp-dir" pod "8499b913-2f91-43f1-bede-5a7462494ef8" (UID: "8499b913-2f91-43f1-bede-5a7462494ef8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:36.560211 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.560167 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-model-cache" pod "8499b913-2f91-43f1-bede-5a7462494ef8" (UID: "8499b913-2f91-43f1-bede-5a7462494ef8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:36.560211 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.560180 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-home" pod "8499b913-2f91-43f1-bede-5a7462494ef8" (UID: "8499b913-2f91-43f1-bede-5a7462494ef8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:36.562710 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.561261 4557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-tmp-dir\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.562710 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.561290 4557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-model-cache\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.562710 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.561304 4557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-home\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.565486 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.563379 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8499b913-2f91-43f1-bede-5a7462494ef8-kube-api-access-9mp4t" pod "8499b913-2f91-43f1-bede-5a7462494ef8" (UID: "8499b913-2f91-43f1-bede-5a7462494ef8"). InnerVolumeSpecName "kube-api-access-9mp4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:36.662575 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.662543 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-tmp-dir\") pod \"c821d166-2220-42fc-801d-62207ca877f2\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " Nov 03 15:46:36.662770 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.662674 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c821d166-2220-42fc-801d-62207ca877f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/c821d166-2220-42fc-801d-62207ca877f2-tls-certs\") pod \"c821d166-2220-42fc-801d-62207ca877f2\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " Nov 03 15:46:36.662842 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.662757 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-dshm\") pod \"8499b913-2f91-43f1-bede-5a7462494ef8\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " Nov 03 15:46:36.662905 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.662877 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-home\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-home\") pod \"c821d166-2220-42fc-801d-62207ca877f2\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " Nov 03 15:46:36.662905 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.662885 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-tmp-dir" pod "c821d166-2220-42fc-801d-62207ca877f2" (UID: "c821d166-2220-42fc-801d-62207ca877f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:36.663008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.662930 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c821d166-2220-42fc-801d-62207ca877f2-kube-api-access-fnnzz\" (UniqueName: \"kubernetes.io/projected/c821d166-2220-42fc-801d-62207ca877f2-kube-api-access-fnnzz\") pod \"c821d166-2220-42fc-801d-62207ca877f2\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " Nov 03 15:46:36.663008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.662989 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8499b913-2f91-43f1-bede-5a7462494ef8-tls-certs\" (UniqueName: \"kubernetes.io/secret/8499b913-2f91-43f1-bede-5a7462494ef8-tls-certs\") pod \"8499b913-2f91-43f1-bede-5a7462494ef8\" (UID: \"8499b913-2f91-43f1-bede-5a7462494ef8\") " Nov 03 15:46:36.663110 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.663027 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-model-cache\") pod \"c821d166-2220-42fc-801d-62207ca877f2\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " Nov 03 15:46:36.663110 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.663062 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-dshm\") pod \"c821d166-2220-42fc-801d-62207ca877f2\" (UID: \"c821d166-2220-42fc-801d-62207ca877f2\") " Nov 03 15:46:36.663207 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.663145 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-home" pod "c821d166-2220-42fc-801d-62207ca877f2" (UID: "c821d166-2220-42fc-801d-62207ca877f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:36.663315 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.663288 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-model-cache" pod "c821d166-2220-42fc-801d-62207ca877f2" (UID: "c821d166-2220-42fc-801d-62207ca877f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:36.663626 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.663601 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mp4t\" (UniqueName: \"kubernetes.io/projected/8499b913-2f91-43f1-bede-5a7462494ef8-kube-api-access-9mp4t\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.663743 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.663631 4557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-home\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.663743 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.663648 4557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-model-cache\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.663743 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.663661 4557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-tmp-dir\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.665520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.665494 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-dshm" pod "c821d166-2220-42fc-801d-62207ca877f2" (UID: "c821d166-2220-42fc-801d-62207ca877f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:36.665520 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.665506 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c821d166-2220-42fc-801d-62207ca877f2-tls-certs" pod "c821d166-2220-42fc-801d-62207ca877f2" (UID: "c821d166-2220-42fc-801d-62207ca877f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:36.665721 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.665506 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-dshm" pod "8499b913-2f91-43f1-bede-5a7462494ef8" (UID: "8499b913-2f91-43f1-bede-5a7462494ef8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:36.665721 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.665672 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8499b913-2f91-43f1-bede-5a7462494ef8-tls-certs" pod "8499b913-2f91-43f1-bede-5a7462494ef8" (UID: "8499b913-2f91-43f1-bede-5a7462494ef8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:36.665901 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.665882 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c821d166-2220-42fc-801d-62207ca877f2-kube-api-access-fnnzz" pod "c821d166-2220-42fc-801d-62207ca877f2" (UID: "c821d166-2220-42fc-801d-62207ca877f2"). InnerVolumeSpecName "kube-api-access-fnnzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:36.736431 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.736390 4557 generic.go:356] "Generic (PLEG): container finished" podID="8499b913-2f91-43f1-bede-5a7462494ef8" containerID="9641131ce77b6edaabf2ee148f3d4857eaf0a2e170668e9ab221df8baf2e7ebf" exitCode=0 Nov 03 15:46:36.736757 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.736454 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" Nov 03 15:46:36.736889 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.736464 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" event={"ID":"8499b913-2f91-43f1-bede-5a7462494ef8","Type":"ContainerDied","Data":"9641131ce77b6edaabf2ee148f3d4857eaf0a2e170668e9ab221df8baf2e7ebf"} Nov 03 15:46:36.736889 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.736852 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45" event={"ID":"8499b913-2f91-43f1-bede-5a7462494ef8","Type":"ContainerDied","Data":"4645221e2cd614642daf8438fefdc2381f9295671866205d67e8fbde3e46fbc8"} Nov 03 15:46:36.737016 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.736899 4557 scope.go:122] "RemoveContainer" containerID="9641131ce77b6edaabf2ee148f3d4857eaf0a2e170668e9ab221df8baf2e7ebf" Nov 03 15:46:36.738859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.738837 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt" Nov 03 15:46:36.738859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.738846 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt" event={"ID":"320d7c16-9625-45ed-bd30-56c0eaf2bbf9","Type":"ContainerDied","Data":"fcf29ba67dbcac324d5be0484cfd668e26faf986ae6b21f422cd74444e3dec47"} Nov 03 15:46:36.739008 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.738943 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcf29ba67dbcac324d5be0484cfd668e26faf986ae6b21f422cd74444e3dec47" Nov 03 15:46:36.740924 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.740901 4557 generic.go:356] "Generic (PLEG): container finished" podID="c821d166-2220-42fc-801d-62207ca877f2" containerID="fb487b339f61a65fa6470bffa59815141496ae641973c69fbbc00e01a6e5507a" exitCode=0 Nov 03 15:46:36.741031 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.740963 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" event={"ID":"c821d166-2220-42fc-801d-62207ca877f2","Type":"ContainerDied","Data":"fb487b339f61a65fa6470bffa59815141496ae641973c69fbbc00e01a6e5507a"} Nov 03 15:46:36.741031 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.740992 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" event={"ID":"c821d166-2220-42fc-801d-62207ca877f2","Type":"ContainerDied","Data":"88df521dc8c0c973f6e7c927679b45d2cae1b2b4572fd8c8bf3bed4ff8d07362"} Nov 03 15:46:36.741031 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.740967 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" Nov 03 15:46:36.748024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.748006 4557 scope.go:122] "RemoveContainer" containerID="9641131ce77b6edaabf2ee148f3d4857eaf0a2e170668e9ab221df8baf2e7ebf" Nov 03 15:46:36.748278 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:46:36.748251 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9641131ce77b6edaabf2ee148f3d4857eaf0a2e170668e9ab221df8baf2e7ebf\": container with ID starting with 9641131ce77b6edaabf2ee148f3d4857eaf0a2e170668e9ab221df8baf2e7ebf not found: ID does not exist" containerID="9641131ce77b6edaabf2ee148f3d4857eaf0a2e170668e9ab221df8baf2e7ebf" Nov 03 15:46:36.748333 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.748281 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9641131ce77b6edaabf2ee148f3d4857eaf0a2e170668e9ab221df8baf2e7ebf"} err="failed to get container status \"9641131ce77b6edaabf2ee148f3d4857eaf0a2e170668e9ab221df8baf2e7ebf\": rpc error: code = NotFound desc = could not find container \"9641131ce77b6edaabf2ee148f3d4857eaf0a2e170668e9ab221df8baf2e7ebf\": container with ID starting with 9641131ce77b6edaabf2ee148f3d4857eaf0a2e170668e9ab221df8baf2e7ebf not found: ID does not exist" Nov 03 15:46:36.748333 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.748304 4557 scope.go:122] "RemoveContainer" containerID="fb487b339f61a65fa6470bffa59815141496ae641973c69fbbc00e01a6e5507a" Nov 03 15:46:36.757179 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.757156 4557 scope.go:122] "RemoveContainer" containerID="fb487b339f61a65fa6470bffa59815141496ae641973c69fbbc00e01a6e5507a" Nov 03 15:46:36.757487 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:46:36.757466 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb487b339f61a65fa6470bffa59815141496ae641973c69fbbc00e01a6e5507a\": container with ID starting with fb487b339f61a65fa6470bffa59815141496ae641973c69fbbc00e01a6e5507a not found: ID does not exist" containerID="fb487b339f61a65fa6470bffa59815141496ae641973c69fbbc00e01a6e5507a" Nov 03 15:46:36.757529 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.757499 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb487b339f61a65fa6470bffa59815141496ae641973c69fbbc00e01a6e5507a"} err="failed to get container status \"fb487b339f61a65fa6470bffa59815141496ae641973c69fbbc00e01a6e5507a\": rpc error: code = NotFound desc = could not find container \"fb487b339f61a65fa6470bffa59815141496ae641973c69fbbc00e01a6e5507a\": container with ID starting with fb487b339f61a65fa6470bffa59815141496ae641973c69fbbc00e01a6e5507a not found: ID does not exist" Nov 03 15:46:36.764861 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.764834 4557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8499b913-2f91-43f1-bede-5a7462494ef8-dshm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.764861 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.764859 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnnzz\" (UniqueName: \"kubernetes.io/projected/c821d166-2220-42fc-801d-62207ca877f2-kube-api-access-fnnzz\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.765000 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.764871 4557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8499b913-2f91-43f1-bede-5a7462494ef8-tls-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.765000 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.764885 4557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c821d166-2220-42fc-801d-62207ca877f2-dshm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.765000 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.764896 4557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c821d166-2220-42fc-801d-62207ca877f2-tls-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:36.951803 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.951766 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45"] Nov 03 15:46:36.961482 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:36.961449 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/distinct-w6-f59d26-kserve-6c6cd598d7-6kb45"] Nov 03 15:46:37.090434 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:37.090398 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv"] Nov 03 15:46:37.102299 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:37.102224 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv"] Nov 03 15:46:37.299827 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:37.299795 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:37.462023 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:37.461909 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f59d26/simulator-w6-f59d26-kserve-5fcb4b4ffc-rt9kv" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 15:46:37.537837 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:37.537806 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:37.663435 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:37.663403 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp"] Nov 03 15:46:37.663639 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:37.663617 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" podUID="3503bbfc-0896-44d5-8b21-02bfa7a360d0" containerName="istio-proxy" containerID="cri-o://2d60679a1424b6925ab6fafb959a6a7eef64509bbc56c11d06667be52b1a3382" gracePeriod=30 Nov 03 15:46:37.841734 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:37.841695 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:38.122114 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.122036 4557 patch_prober.go:29] interesting pod/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:46:38.122114 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.122091 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" podUID="3503bbfc-0896-44d5-8b21-02bfa7a360d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:46:38.280316 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.280286 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:38.285486 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.285460 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" path="/var/lib/kubelet/pods/8499b913-2f91-43f1-bede-5a7462494ef8/volumes" Nov 03 15:46:38.285937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.285923 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c821d166-2220-42fc-801d-62207ca877f2" path="/var/lib/kubelet/pods/c821d166-2220-42fc-801d-62207ca877f2/volumes" Nov 03 15:46:38.502653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.500795 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:38.847109 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.847080 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj"] Nov 03 15:46:38.847542 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.847529 4557 cpu_manager.go:414] "containerMap: removing container" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" Nov 03 15:46:38.847636 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.847543 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" Nov 03 15:46:38.847636 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.847562 4557 cpu_manager.go:414] "containerMap: removing container" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" Nov 03 15:46:38.847636 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.847568 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" Nov 03 15:46:38.847636 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.847596 4557 cpu_manager.go:414] "containerMap: removing container" podUID="320d7c16-9625-45ed-bd30-56c0eaf2bbf9" containerName="revoke-keys" Nov 03 15:46:38.847636 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.847602 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320d7c16-9625-45ed-bd30-56c0eaf2bbf9" containerName="revoke-keys" Nov 03 15:46:38.847830 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.847682 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="8499b913-2f91-43f1-bede-5a7462494ef8" containerName="main" Nov 03 15:46:38.847830 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.847692 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="320d7c16-9625-45ed-bd30-56c0eaf2bbf9" containerName="revoke-keys" Nov 03 15:46:38.847830 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.847702 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="c821d166-2220-42fc-801d-62207ca877f2" containerName="main" Nov 03 15:46:38.850816 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.850794 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj" Nov 03 15:46:38.914944 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.914915 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj"] Nov 03 15:46:38.960865 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.960839 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:46:38.961036 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.960916 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:46:38.998074 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.998046 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33449900-1ed0-43f1-88d0-422e2d5b56a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj\" (UID: \"33449900-1ed0-43f1-88d0-422e2d5b56a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj" Nov 03 15:46:38.998238 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.998080 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68vvk\" (UniqueName: \"kubernetes.io/projected/33449900-1ed0-43f1-88d0-422e2d5b56a2-kube-api-access-68vvk\") pod \"maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj\" (UID: \"33449900-1ed0-43f1-88d0-422e2d5b56a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj" Nov 03 15:46:38.998238 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:38.998172 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33449900-1ed0-43f1-88d0-422e2d5b56a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj\" (UID: \"33449900-1ed0-43f1-88d0-422e2d5b56a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj" Nov 03 15:46:39.099879 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:39.099790 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33449900-1ed0-43f1-88d0-422e2d5b56a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj\" (UID: \"33449900-1ed0-43f1-88d0-422e2d5b56a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj" Nov 03 15:46:39.099879 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:39.099831 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vvk\" (UniqueName: \"kubernetes.io/projected/33449900-1ed0-43f1-88d0-422e2d5b56a2-kube-api-access-68vvk\") pod \"maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj\" (UID: \"33449900-1ed0-43f1-88d0-422e2d5b56a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj" Nov 03 15:46:39.100084 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:39.099884 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33449900-1ed0-43f1-88d0-422e2d5b56a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj\" (UID: \"33449900-1ed0-43f1-88d0-422e2d5b56a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj" Nov 03 15:46:39.100556 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:39.100528 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33449900-1ed0-43f1-88d0-422e2d5b56a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj\" (UID: \"33449900-1ed0-43f1-88d0-422e2d5b56a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj" Nov 03 15:46:39.102107 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:39.102079 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33449900-1ed0-43f1-88d0-422e2d5b56a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj\" (UID: \"33449900-1ed0-43f1-88d0-422e2d5b56a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj" Nov 03 15:46:39.164567 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:39.164536 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68vvk\" (UniqueName: \"kubernetes.io/projected/33449900-1ed0-43f1-88d0-422e2d5b56a2-kube-api-access-68vvk\") pod \"maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj\" (UID: \"33449900-1ed0-43f1-88d0-422e2d5b56a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj" Nov 03 15:46:39.463517 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:39.463431 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj" Nov 03 15:46:39.611048 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:39.611004 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:39.842815 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:39.842790 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj"] Nov 03 15:46:39.844729 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:46:39.844700 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33449900_1ed0_43f1_88d0_422e2d5b56a2.slice/crio-21927e7cbdb677dc1eaf7080c95ad77327f5479bbfcc09e496356228cd708c3d WatchSource:0}: Error finding container 21927e7cbdb677dc1eaf7080c95ad77327f5479bbfcc09e496356228cd708c3d: Status 404 returned error can't find the container with id 21927e7cbdb677dc1eaf7080c95ad77327f5479bbfcc09e496356228cd708c3d Nov 03 15:46:40.672756 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:40.671079 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:40.771380 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:40.771346 4557 generic.go:356] "Generic (PLEG): container finished" podID="33449900-1ed0-43f1-88d0-422e2d5b56a2" containerID="4f6ac07c9dd0c4ee551e374fc64eb56811b7482b166fa4e4a63f2369297cd7f9" exitCode=0 Nov 03 15:46:40.771530 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:40.771398 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj" event={"ID":"33449900-1ed0-43f1-88d0-422e2d5b56a2","Type":"ContainerDied","Data":"4f6ac07c9dd0c4ee551e374fc64eb56811b7482b166fa4e4a63f2369297cd7f9"} Nov 03 15:46:40.771530 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:40.771452 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj" event={"ID":"33449900-1ed0-43f1-88d0-422e2d5b56a2","Type":"ContainerStarted","Data":"21927e7cbdb677dc1eaf7080c95ad77327f5479bbfcc09e496356228cd708c3d"} Nov 03 15:46:40.944302 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:40.944211 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:41.830728 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:41.830679 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:41.948808 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:41.948782 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj" Nov 03 15:46:42.033012 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.032922 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33449900-1ed0-43f1-88d0-422e2d5b56a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33449900-1ed0-43f1-88d0-422e2d5b56a2-maas-api-openshift-ca\") pod \"33449900-1ed0-43f1-88d0-422e2d5b56a2\" (UID: \"33449900-1ed0-43f1-88d0-422e2d5b56a2\") " Nov 03 15:46:42.033182 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.033010 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33449900-1ed0-43f1-88d0-422e2d5b56a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33449900-1ed0-43f1-88d0-422e2d5b56a2-maas-api-xks-ca\") pod \"33449900-1ed0-43f1-88d0-422e2d5b56a2\" (UID: \"33449900-1ed0-43f1-88d0-422e2d5b56a2\") " Nov 03 15:46:42.033182 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.033042 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33449900-1ed0-43f1-88d0-422e2d5b56a2-kube-api-access-68vvk\" (UniqueName: \"kubernetes.io/projected/33449900-1ed0-43f1-88d0-422e2d5b56a2-kube-api-access-68vvk\") pod \"33449900-1ed0-43f1-88d0-422e2d5b56a2\" (UID: \"33449900-1ed0-43f1-88d0-422e2d5b56a2\") " Nov 03 15:46:42.033327 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.033298 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33449900-1ed0-43f1-88d0-422e2d5b56a2-maas-api-openshift-ca" pod "33449900-1ed0-43f1-88d0-422e2d5b56a2" (UID: "33449900-1ed0-43f1-88d0-422e2d5b56a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:42.033610 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.033574 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33449900-1ed0-43f1-88d0-422e2d5b56a2-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:42.035157 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.035129 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33449900-1ed0-43f1-88d0-422e2d5b56a2-kube-api-access-68vvk" pod "33449900-1ed0-43f1-88d0-422e2d5b56a2" (UID: "33449900-1ed0-43f1-88d0-422e2d5b56a2"). InnerVolumeSpecName "kube-api-access-68vvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:42.035157 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.035135 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33449900-1ed0-43f1-88d0-422e2d5b56a2-maas-api-xks-ca" pod "33449900-1ed0-43f1-88d0-422e2d5b56a2" (UID: "33449900-1ed0-43f1-88d0-422e2d5b56a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:42.134712 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.134669 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33449900-1ed0-43f1-88d0-422e2d5b56a2-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:42.134712 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.134706 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68vvk\" (UniqueName: \"kubernetes.io/projected/33449900-1ed0-43f1-88d0-422e2d5b56a2-kube-api-access-68vvk\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:42.371175 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.371133 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:42.654019 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.653981 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:42.713461 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.713418 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb"] Nov 03 15:46:42.713657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.713636 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" podUID="acfc1375-2b70-4aec-b161-4e24c39a8d59" containerName="istio-proxy" containerID="cri-o://15cc961e3e3f64b185896db9d129d4d27f921b1c188aa480feaa7ec359b0ac2c" gracePeriod=30 Nov 03 15:46:42.726462 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.726433 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp"] Nov 03 15:46:42.726655 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.726634 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" podUID="2a6c2b03-8f2a-47cd-8e66-923505d511d7" containerName="istio-proxy" containerID="cri-o://c0eccdc48adfeedf19b06d2e30b748a2704e88b8c55d8b15df46fdf90dcf3c9d" gracePeriod=30 Nov 03 15:46:42.732629 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.732605 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd"] Nov 03 15:46:42.732788 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.732771 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" podUID="327437b8-3b27-4490-9cf6-d965799a1c3a" containerName="istio-proxy" containerID="cri-o://6b7f1e2478ece1d7bc4c704c57719f10fc9f762c68a735aa04c0b1e7e7f5606d" gracePeriod=30 Nov 03 15:46:42.841449 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.841403 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj" event={"ID":"33449900-1ed0-43f1-88d0-422e2d5b56a2","Type":"ContainerDied","Data":"21927e7cbdb677dc1eaf7080c95ad77327f5479bbfcc09e496356228cd708c3d"} Nov 03 15:46:42.841928 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.841472 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21927e7cbdb677dc1eaf7080c95ad77327f5479bbfcc09e496356228cd708c3d" Nov 03 15:46:42.841928 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.841544 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj" Nov 03 15:46:42.854832 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:42.850106 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:43.019957 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.019930 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:46:43.152966 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.152928 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istiod-ca-cert\") pod \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " Nov 03 15:46:43.153134 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.152983 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-data\") pod \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " Nov 03 15:46:43.153134 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.153044 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-envoy\") pod \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " Nov 03 15:46:43.153134 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.153109 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-workload-certs\") pod \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " Nov 03 15:46:43.153300 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.153183 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-token\" (UniqueName: \"kubernetes.io/projected/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-token\") pod \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " Nov 03 15:46:43.153300 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.153231 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3503bbfc-0896-44d5-8b21-02bfa7a360d0-kube-api-access-rmdlq\" (UniqueName: \"kubernetes.io/projected/3503bbfc-0896-44d5-8b21-02bfa7a360d0-kube-api-access-rmdlq\") pod \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " Nov 03 15:46:43.153300 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.153266 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-podinfo\") pod \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " Nov 03 15:46:43.153463 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.153330 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-workload-socket\") pod \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " Nov 03 15:46:43.153463 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.153339 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istiod-ca-cert" pod "3503bbfc-0896-44d5-8b21-02bfa7a360d0" (UID: "3503bbfc-0896-44d5-8b21-02bfa7a360d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:43.153463 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.153402 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-credential-socket\") pod \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\" (UID: \"3503bbfc-0896-44d5-8b21-02bfa7a360d0\") " Nov 03 15:46:43.153463 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.153438 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-workload-certs" pod "3503bbfc-0896-44d5-8b21-02bfa7a360d0" (UID: "3503bbfc-0896-44d5-8b21-02bfa7a360d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:43.153711 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.153536 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-data" pod "3503bbfc-0896-44d5-8b21-02bfa7a360d0" (UID: "3503bbfc-0896-44d5-8b21-02bfa7a360d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:43.153711 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.153665 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-workload-socket" pod "3503bbfc-0896-44d5-8b21-02bfa7a360d0" (UID: "3503bbfc-0896-44d5-8b21-02bfa7a360d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:43.153822 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.153746 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-credential-socket" pod "3503bbfc-0896-44d5-8b21-02bfa7a360d0" (UID: "3503bbfc-0896-44d5-8b21-02bfa7a360d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:43.153960 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.153944 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:43.154032 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.153964 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:43.154032 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.153979 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:43.154032 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.153994 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:43.154032 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.154007 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:43.155556 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.155531 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-envoy" pod "3503bbfc-0896-44d5-8b21-02bfa7a360d0" (UID: "3503bbfc-0896-44d5-8b21-02bfa7a360d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:43.155693 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.155536 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-token" pod "3503bbfc-0896-44d5-8b21-02bfa7a360d0" (UID: "3503bbfc-0896-44d5-8b21-02bfa7a360d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:43.155834 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.155816 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-podinfo" pod "3503bbfc-0896-44d5-8b21-02bfa7a360d0" (UID: "3503bbfc-0896-44d5-8b21-02bfa7a360d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:46:43.155902 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.155860 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3503bbfc-0896-44d5-8b21-02bfa7a360d0-kube-api-access-rmdlq" pod "3503bbfc-0896-44d5-8b21-02bfa7a360d0" (UID: "3503bbfc-0896-44d5-8b21-02bfa7a360d0"). InnerVolumeSpecName "kube-api-access-rmdlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:43.255828 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.255747 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:43.255828 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.255778 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:43.255828 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.255792 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmdlq\" (UniqueName: \"kubernetes.io/projected/3503bbfc-0896-44d5-8b21-02bfa7a360d0-kube-api-access-rmdlq\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:43.255828 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.255806 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3503bbfc-0896-44d5-8b21-02bfa7a360d0-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:43.300649 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.300614 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7"] Nov 03 15:46:43.321299 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.321264 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs"] Nov 03 15:46:43.353845 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.353815 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:43.575309 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.575273 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:43.786013 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.785980 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf"] Nov 03 15:46:43.786509 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.786489 4557 cpu_manager.go:414] "containerMap: removing container" podUID="3503bbfc-0896-44d5-8b21-02bfa7a360d0" containerName="istio-proxy" Nov 03 15:46:43.786646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.786510 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3503bbfc-0896-44d5-8b21-02bfa7a360d0" containerName="istio-proxy" Nov 03 15:46:43.786646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.786537 4557 cpu_manager.go:414] "containerMap: removing container" podUID="33449900-1ed0-43f1-88d0-422e2d5b56a2" containerName="revoke-keys" Nov 03 15:46:43.786646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.786547 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33449900-1ed0-43f1-88d0-422e2d5b56a2" containerName="revoke-keys" Nov 03 15:46:43.786646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.786570 4557 cpu_manager.go:414] "containerMap: removing container" podUID="3503bbfc-0896-44d5-8b21-02bfa7a360d0" containerName="istio-proxy" Nov 03 15:46:43.786646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.786610 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3503bbfc-0896-44d5-8b21-02bfa7a360d0" containerName="istio-proxy" Nov 03 15:46:43.786911 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.786717 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="3503bbfc-0896-44d5-8b21-02bfa7a360d0" containerName="istio-proxy" Nov 03 15:46:43.786911 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.786733 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="33449900-1ed0-43f1-88d0-422e2d5b56a2" containerName="revoke-keys" Nov 03 15:46:43.790169 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.790146 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf" Nov 03 15:46:43.862504 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.862421 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf"] Nov 03 15:46:43.864247 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.864211 4557 generic.go:356] "Generic (PLEG): container finished" podID="3503bbfc-0896-44d5-8b21-02bfa7a360d0" containerID="2d60679a1424b6925ab6fafb959a6a7eef64509bbc56c11d06667be52b1a3382" exitCode=0 Nov 03 15:46:43.864404 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.864282 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" Nov 03 15:46:43.864404 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.864289 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" event={"ID":"3503bbfc-0896-44d5-8b21-02bfa7a360d0","Type":"ContainerDied","Data":"2d60679a1424b6925ab6fafb959a6a7eef64509bbc56c11d06667be52b1a3382"} Nov 03 15:46:43.864404 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.864331 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp" event={"ID":"3503bbfc-0896-44d5-8b21-02bfa7a360d0","Type":"ContainerDied","Data":"5f8c17ca17f905187796060ed816b85e2624d577fa6d49e8f32b8caee33a4ebe"} Nov 03 15:46:43.864404 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.864353 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tf5f\" (UniqueName: \"kubernetes.io/projected/2ac71731-3b40-418b-8ddf-816a8aafd5e7-kube-api-access-6tf5f\") pod \"maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf\" (UID: \"2ac71731-3b40-418b-8ddf-816a8aafd5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf" Nov 03 15:46:43.864404 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.864360 4557 scope.go:122] "RemoveContainer" containerID="2d60679a1424b6925ab6fafb959a6a7eef64509bbc56c11d06667be52b1a3382" Nov 03 15:46:43.864715 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.864419 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac71731-3b40-418b-8ddf-816a8aafd5e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf\" (UID: \"2ac71731-3b40-418b-8ddf-816a8aafd5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf" Nov 03 15:46:43.864715 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.864482 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac71731-3b40-418b-8ddf-816a8aafd5e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf\" (UID: \"2ac71731-3b40-418b-8ddf-816a8aafd5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf" Nov 03 15:46:43.880341 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.880314 4557 scope.go:122] "RemoveContainer" containerID="32b07a3fb046af46137f5f70e6ff769b295c9b215dda6d2710ba87886d76ec5d" Nov 03 15:46:43.892244 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.892224 4557 scope.go:122] "RemoveContainer" containerID="2d60679a1424b6925ab6fafb959a6a7eef64509bbc56c11d06667be52b1a3382" Nov 03 15:46:43.892711 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:46:43.892574 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d60679a1424b6925ab6fafb959a6a7eef64509bbc56c11d06667be52b1a3382\": container with ID starting with 2d60679a1424b6925ab6fafb959a6a7eef64509bbc56c11d06667be52b1a3382 not found: ID does not exist" containerID="2d60679a1424b6925ab6fafb959a6a7eef64509bbc56c11d06667be52b1a3382" Nov 03 15:46:43.892711 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.892641 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d60679a1424b6925ab6fafb959a6a7eef64509bbc56c11d06667be52b1a3382"} err="failed to get container status \"2d60679a1424b6925ab6fafb959a6a7eef64509bbc56c11d06667be52b1a3382\": rpc error: code = NotFound desc = could not find container \"2d60679a1424b6925ab6fafb959a6a7eef64509bbc56c11d06667be52b1a3382\": container with ID starting with 2d60679a1424b6925ab6fafb959a6a7eef64509bbc56c11d06667be52b1a3382 not found: ID does not exist" Nov 03 15:46:43.892711 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.892667 4557 scope.go:122] "RemoveContainer" containerID="32b07a3fb046af46137f5f70e6ff769b295c9b215dda6d2710ba87886d76ec5d" Nov 03 15:46:43.893070 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:46:43.893043 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b07a3fb046af46137f5f70e6ff769b295c9b215dda6d2710ba87886d76ec5d\": container with ID starting with 32b07a3fb046af46137f5f70e6ff769b295c9b215dda6d2710ba87886d76ec5d not found: ID does not exist" containerID="32b07a3fb046af46137f5f70e6ff769b295c9b215dda6d2710ba87886d76ec5d" Nov 03 15:46:43.893142 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.893079 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b07a3fb046af46137f5f70e6ff769b295c9b215dda6d2710ba87886d76ec5d"} err="failed to get container status \"32b07a3fb046af46137f5f70e6ff769b295c9b215dda6d2710ba87886d76ec5d\": rpc error: code = NotFound desc = could not find container \"32b07a3fb046af46137f5f70e6ff769b295c9b215dda6d2710ba87886d76ec5d\": container with ID starting with 32b07a3fb046af46137f5f70e6ff769b295c9b215dda6d2710ba87886d76ec5d not found: ID does not exist" Nov 03 15:46:43.927882 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.927853 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:46:43.928044 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.928028 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:46:43.965562 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.965531 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac71731-3b40-418b-8ddf-816a8aafd5e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf\" (UID: \"2ac71731-3b40-418b-8ddf-816a8aafd5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf" Nov 03 15:46:43.965766 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.965734 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tf5f\" (UniqueName: \"kubernetes.io/projected/2ac71731-3b40-418b-8ddf-816a8aafd5e7-kube-api-access-6tf5f\") pod \"maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf\" (UID: \"2ac71731-3b40-418b-8ddf-816a8aafd5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf" Nov 03 15:46:43.965841 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.965776 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac71731-3b40-418b-8ddf-816a8aafd5e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf\" (UID: \"2ac71731-3b40-418b-8ddf-816a8aafd5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf" Nov 03 15:46:43.966391 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.966368 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac71731-3b40-418b-8ddf-816a8aafd5e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf\" (UID: \"2ac71731-3b40-418b-8ddf-816a8aafd5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf" Nov 03 15:46:43.967854 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:43.967824 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac71731-3b40-418b-8ddf-816a8aafd5e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf\" (UID: \"2ac71731-3b40-418b-8ddf-816a8aafd5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf" Nov 03 15:46:44.049057 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:44.049021 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp"] Nov 03 15:46:44.053020 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:44.052992 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tf5f\" (UniqueName: \"kubernetes.io/projected/2ac71731-3b40-418b-8ddf-816a8aafd5e7-kube-api-access-6tf5f\") pod \"maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf\" (UID: \"2ac71731-3b40-418b-8ddf-816a8aafd5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf" Nov 03 15:46:44.057256 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:44.057233 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-78034f-openshift-default-dd596bfbf-h8vtp"] Nov 03 15:46:44.103898 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:44.103859 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf" Nov 03 15:46:44.285801 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:44.285770 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3503bbfc-0896-44d5-8b21-02bfa7a360d0" path="/var/lib/kubelet/pods/3503bbfc-0896-44d5-8b21-02bfa7a360d0/volumes" Nov 03 15:46:44.489849 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:44.489823 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf"] Nov 03 15:46:44.491019 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:46:44.490988 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac71731_3b40_418b_8ddf_816a8aafd5e7.slice/crio-7a5aa3d339a0602c22d29c9127752220d0e684d15d102c7ba87141ebd28e6d64 WatchSource:0}: Error finding container 7a5aa3d339a0602c22d29c9127752220d0e684d15d102c7ba87141ebd28e6d64: Status 404 returned error can't find the container with id 7a5aa3d339a0602c22d29c9127752220d0e684d15d102c7ba87141ebd28e6d64 Nov 03 15:46:44.873557 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:44.873522 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf" event={"ID":"2ac71731-3b40-418b-8ddf-816a8aafd5e7","Type":"ContainerStarted","Data":"071e23befcc092ef8e1c669a76b5aee215ab69fcdc23f60d7928bdb4aadd664c"} Nov 03 15:46:44.874050 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:44.873571 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf" event={"ID":"2ac71731-3b40-418b-8ddf-816a8aafd5e7","Type":"ContainerStarted","Data":"7a5aa3d339a0602c22d29c9127752220d0e684d15d102c7ba87141ebd28e6d64"} Nov 03 15:46:45.008535 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:45.008471 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf" podStartSLOduration=2.008451912 podStartE2EDuration="2.008451912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:46:45.005252438 +0000 UTC m=+2314.848373980" watchObservedRunningTime="2026-11-03 15:46:45.008451912 +0000 UTC m=+2314.851573458" Nov 03 15:46:45.705498 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:45.705326 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:45.881699 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:45.881665 4557 generic.go:356] "Generic (PLEG): container finished" podID="2ac71731-3b40-418b-8ddf-816a8aafd5e7" containerID="071e23befcc092ef8e1c669a76b5aee215ab69fcdc23f60d7928bdb4aadd664c" exitCode=0 Nov 03 15:46:45.882130 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:45.881751 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf" event={"ID":"2ac71731-3b40-418b-8ddf-816a8aafd5e7","Type":"ContainerDied","Data":"071e23befcc092ef8e1c669a76b5aee215ab69fcdc23f60d7928bdb4aadd664c"} Nov 03 15:46:46.530311 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:46:46.530273 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:46:46.861810 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:46.861758 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:47.044965 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.044934 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:47.047945 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.047921 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf" Nov 03 15:46:47.204061 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.203958 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ac71731-3b40-418b-8ddf-816a8aafd5e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac71731-3b40-418b-8ddf-816a8aafd5e7-maas-api-openshift-ca\") pod \"2ac71731-3b40-418b-8ddf-816a8aafd5e7\" (UID: \"2ac71731-3b40-418b-8ddf-816a8aafd5e7\") " Nov 03 15:46:47.204238 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.204159 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac71731-3b40-418b-8ddf-816a8aafd5e7-kube-api-access-6tf5f\" (UniqueName: \"kubernetes.io/projected/2ac71731-3b40-418b-8ddf-816a8aafd5e7-kube-api-access-6tf5f\") pod \"2ac71731-3b40-418b-8ddf-816a8aafd5e7\" (UID: \"2ac71731-3b40-418b-8ddf-816a8aafd5e7\") " Nov 03 15:46:47.204314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.204246 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ac71731-3b40-418b-8ddf-816a8aafd5e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac71731-3b40-418b-8ddf-816a8aafd5e7-maas-api-xks-ca\") pod \"2ac71731-3b40-418b-8ddf-816a8aafd5e7\" (UID: \"2ac71731-3b40-418b-8ddf-816a8aafd5e7\") " Nov 03 15:46:47.204314 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.204263 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac71731-3b40-418b-8ddf-816a8aafd5e7-maas-api-openshift-ca" pod "2ac71731-3b40-418b-8ddf-816a8aafd5e7" (UID: "2ac71731-3b40-418b-8ddf-816a8aafd5e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:47.204792 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.204759 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac71731-3b40-418b-8ddf-816a8aafd5e7-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:47.206853 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.206821 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac71731-3b40-418b-8ddf-816a8aafd5e7-maas-api-xks-ca" pod "2ac71731-3b40-418b-8ddf-816a8aafd5e7" (UID: "2ac71731-3b40-418b-8ddf-816a8aafd5e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:47.206943 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.206869 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac71731-3b40-418b-8ddf-816a8aafd5e7-kube-api-access-6tf5f" pod "2ac71731-3b40-418b-8ddf-816a8aafd5e7" (UID: "2ac71731-3b40-418b-8ddf-816a8aafd5e7"). InnerVolumeSpecName "kube-api-access-6tf5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:47.306092 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.306054 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tf5f\" (UniqueName: \"kubernetes.io/projected/2ac71731-3b40-418b-8ddf-816a8aafd5e7-kube-api-access-6tf5f\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:47.306092 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.306086 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac71731-3b40-418b-8ddf-816a8aafd5e7-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:47.897569 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.897534 4557 generic.go:356] "Generic (PLEG): container finished" podID="2a6c2b03-8f2a-47cd-8e66-923505d511d7" containerID="c0eccdc48adfeedf19b06d2e30b748a2704e88b8c55d8b15df46fdf90dcf3c9d" exitCode=0 Nov 03 15:46:47.897771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.897618 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" event={"ID":"2a6c2b03-8f2a-47cd-8e66-923505d511d7","Type":"ContainerDied","Data":"c0eccdc48adfeedf19b06d2e30b748a2704e88b8c55d8b15df46fdf90dcf3c9d"} Nov 03 15:46:47.912609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.899667 4557 generic.go:356] "Generic (PLEG): container finished" podID="327437b8-3b27-4490-9cf6-d965799a1c3a" containerID="6b7f1e2478ece1d7bc4c704c57719f10fc9f762c68a735aa04c0b1e7e7f5606d" exitCode=0 Nov 03 15:46:47.912609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.899764 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" event={"ID":"327437b8-3b27-4490-9cf6-d965799a1c3a","Type":"ContainerDied","Data":"6b7f1e2478ece1d7bc4c704c57719f10fc9f762c68a735aa04c0b1e7e7f5606d"} Nov 03 15:46:47.912609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.901545 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf" event={"ID":"2ac71731-3b40-418b-8ddf-816a8aafd5e7","Type":"ContainerDied","Data":"7a5aa3d339a0602c22d29c9127752220d0e684d15d102c7ba87141ebd28e6d64"} Nov 03 15:46:47.912609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.901625 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a5aa3d339a0602c22d29c9127752220d0e684d15d102c7ba87141ebd28e6d64" Nov 03 15:46:47.912609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.901705 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf" Nov 03 15:46:47.912609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.910359 4557 generic.go:356] "Generic (PLEG): container finished" podID="acfc1375-2b70-4aec-b161-4e24c39a8d59" containerID="15cc961e3e3f64b185896db9d129d4d27f921b1c188aa480feaa7ec359b0ac2c" exitCode=0 Nov 03 15:46:47.912609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.910464 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" event={"ID":"acfc1375-2b70-4aec-b161-4e24c39a8d59","Type":"ContainerDied","Data":"15cc961e3e3f64b185896db9d129d4d27f921b1c188aa480feaa7ec359b0ac2c"} Nov 03 15:46:47.995413 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:47.995380 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:46:48.068724 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.068698 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:46:48.071712 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.071692 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:46:48.115623 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.115561 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-workload-socket\") pod \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " Nov 03 15:46:48.115781 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.115645 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-credential-socket\") pod \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " Nov 03 15:46:48.115863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.115822 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-data\") pod \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " Nov 03 15:46:48.115863 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.115828 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-workload-socket" pod "2a6c2b03-8f2a-47cd-8e66-923505d511d7" (UID: "2a6c2b03-8f2a-47cd-8e66-923505d511d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.115945 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.115896 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istiod-ca-cert\") pod \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " Nov 03 15:46:48.115979 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.115947 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a6c2b03-8f2a-47cd-8e66-923505d511d7-kube-api-access-g24pm\" (UniqueName: \"kubernetes.io/projected/2a6c2b03-8f2a-47cd-8e66-923505d511d7-kube-api-access-g24pm\") pod \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " Nov 03 15:46:48.116024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.115974 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-credential-socket" pod "2a6c2b03-8f2a-47cd-8e66-923505d511d7" (UID: "2a6c2b03-8f2a-47cd-8e66-923505d511d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.116024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.116007 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-data" pod "2a6c2b03-8f2a-47cd-8e66-923505d511d7" (UID: "2a6c2b03-8f2a-47cd-8e66-923505d511d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.116024 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.116014 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-podinfo\") pod \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " Nov 03 15:46:48.116132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.116103 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-token\" (UniqueName: \"kubernetes.io/projected/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-token\") pod \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " Nov 03 15:46:48.116178 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.116160 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-envoy\") pod \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " Nov 03 15:46:48.116217 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.116191 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-workload-certs\") pod \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\" (UID: \"2a6c2b03-8f2a-47cd-8e66-923505d511d7\") " Nov 03 15:46:48.116260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.116227 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istiod-ca-cert" pod "2a6c2b03-8f2a-47cd-8e66-923505d511d7" (UID: "2a6c2b03-8f2a-47cd-8e66-923505d511d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:48.116729 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.116695 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-workload-certs" pod "2a6c2b03-8f2a-47cd-8e66-923505d511d7" (UID: "2a6c2b03-8f2a-47cd-8e66-923505d511d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.116869 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.116779 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.116869 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.116803 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.116869 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.116819 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.116869 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.116834 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.118167 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.118131 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-podinfo" pod "2a6c2b03-8f2a-47cd-8e66-923505d511d7" (UID: "2a6c2b03-8f2a-47cd-8e66-923505d511d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:46:48.118167 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.118142 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6c2b03-8f2a-47cd-8e66-923505d511d7-kube-api-access-g24pm" pod "2a6c2b03-8f2a-47cd-8e66-923505d511d7" (UID: "2a6c2b03-8f2a-47cd-8e66-923505d511d7"). InnerVolumeSpecName "kube-api-access-g24pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:48.118319 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.118204 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-token" pod "2a6c2b03-8f2a-47cd-8e66-923505d511d7" (UID: "2a6c2b03-8f2a-47cd-8e66-923505d511d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:48.118319 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.118250 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-envoy" pod "2a6c2b03-8f2a-47cd-8e66-923505d511d7" (UID: "2a6c2b03-8f2a-47cd-8e66-923505d511d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.217707 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.217676 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327437b8-3b27-4490-9cf6-d965799a1c3a-kube-api-access-d78dz\" (UniqueName: \"kubernetes.io/projected/327437b8-3b27-4490-9cf6-d965799a1c3a-kube-api-access-d78dz\") pod \"327437b8-3b27-4490-9cf6-d965799a1c3a\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " Nov 03 15:46:48.217888 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.217717 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-workload-socket\") pod \"acfc1375-2b70-4aec-b161-4e24c39a8d59\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " Nov 03 15:46:48.217888 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.217772 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acfc1375-2b70-4aec-b161-4e24c39a8d59-kube-api-access-87cv5\" (UniqueName: \"kubernetes.io/projected/acfc1375-2b70-4aec-b161-4e24c39a8d59-kube-api-access-87cv5\") pod \"acfc1375-2b70-4aec-b161-4e24c39a8d59\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " Nov 03 15:46:48.217888 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.217806 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-data\") pod \"acfc1375-2b70-4aec-b161-4e24c39a8d59\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " Nov 03 15:46:48.217888 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.217830 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-token\" (UniqueName: \"kubernetes.io/projected/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-token\") pod \"acfc1375-2b70-4aec-b161-4e24c39a8d59\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " Nov 03 15:46:48.217888 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.217863 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-data\") pod \"327437b8-3b27-4490-9cf6-d965799a1c3a\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " Nov 03 15:46:48.218176 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.217900 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-token\" (UniqueName: \"kubernetes.io/projected/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-token\") pod \"327437b8-3b27-4490-9cf6-d965799a1c3a\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " Nov 03 15:46:48.218176 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.217932 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-podinfo\") pod \"327437b8-3b27-4490-9cf6-d965799a1c3a\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " Nov 03 15:46:48.218176 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.218014 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acfc1375-2b70-4aec-b161-4e24c39a8d59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acfc1375-2b70-4aec-b161-4e24c39a8d59-istiod-ca-cert\") pod \"acfc1375-2b70-4aec-b161-4e24c39a8d59\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " Nov 03 15:46:48.218176 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.218047 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-workload-certs\") pod \"327437b8-3b27-4490-9cf6-d965799a1c3a\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " Nov 03 15:46:48.218176 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.218093 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-credential-socket\") pod \"327437b8-3b27-4490-9cf6-d965799a1c3a\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " Nov 03 15:46:48.218176 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.218100 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-workload-socket" pod "acfc1375-2b70-4aec-b161-4e24c39a8d59" (UID: "acfc1375-2b70-4aec-b161-4e24c39a8d59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.218176 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.218151 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-workload-socket\") pod \"327437b8-3b27-4490-9cf6-d965799a1c3a\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " Nov 03 15:46:48.218566 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.218212 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-podinfo\") pod \"acfc1375-2b70-4aec-b161-4e24c39a8d59\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " Nov 03 15:46:48.218566 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.218239 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-envoy\") pod \"327437b8-3b27-4490-9cf6-d965799a1c3a\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " Nov 03 15:46:48.218566 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.218306 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/327437b8-3b27-4490-9cf6-d965799a1c3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/327437b8-3b27-4490-9cf6-d965799a1c3a-istiod-ca-cert\") pod \"327437b8-3b27-4490-9cf6-d965799a1c3a\" (UID: \"327437b8-3b27-4490-9cf6-d965799a1c3a\") " Nov 03 15:46:48.218566 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.218309 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-data" pod "327437b8-3b27-4490-9cf6-d965799a1c3a" (UID: "327437b8-3b27-4490-9cf6-d965799a1c3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.218566 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.218338 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-envoy\") pod \"acfc1375-2b70-4aec-b161-4e24c39a8d59\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " Nov 03 15:46:48.218566 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.218366 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-workload-certs\") pod \"acfc1375-2b70-4aec-b161-4e24c39a8d59\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " Nov 03 15:46:48.218566 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.218407 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-credential-socket\") pod \"acfc1375-2b70-4aec-b161-4e24c39a8d59\" (UID: \"acfc1375-2b70-4aec-b161-4e24c39a8d59\") " Nov 03 15:46:48.218566 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.218462 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-credential-socket" pod "327437b8-3b27-4490-9cf6-d965799a1c3a" (UID: "327437b8-3b27-4490-9cf6-d965799a1c3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.219004 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.218688 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-workload-certs" pod "327437b8-3b27-4490-9cf6-d965799a1c3a" (UID: "327437b8-3b27-4490-9cf6-d965799a1c3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.219004 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.218717 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acfc1375-2b70-4aec-b161-4e24c39a8d59-istiod-ca-cert" pod "acfc1375-2b70-4aec-b161-4e24c39a8d59" (UID: "acfc1375-2b70-4aec-b161-4e24c39a8d59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:48.219004 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.218879 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-credential-socket" pod "acfc1375-2b70-4aec-b161-4e24c39a8d59" (UID: "acfc1375-2b70-4aec-b161-4e24c39a8d59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.219183 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.219059 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-workload-socket" pod "327437b8-3b27-4490-9cf6-d965799a1c3a" (UID: "327437b8-3b27-4490-9cf6-d965799a1c3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.219546 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.219502 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-workload-certs" pod "acfc1375-2b70-4aec-b161-4e24c39a8d59" (UID: "acfc1375-2b70-4aec-b161-4e24c39a8d59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.219737 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.219714 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.219811 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.219757 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g24pm\" (UniqueName: \"kubernetes.io/projected/2a6c2b03-8f2a-47cd-8e66-923505d511d7-kube-api-access-g24pm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.219811 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.219774 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.219811 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.219789 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.219811 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.219804 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.220075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.219820 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.220075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.219834 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a6c2b03-8f2a-47cd-8e66-923505d511d7-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.220075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.219863 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.220075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.219877 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acfc1375-2b70-4aec-b161-4e24c39a8d59-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.220075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.219892 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.220075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.219907 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.220075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.219929 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/327437b8-3b27-4490-9cf6-d965799a1c3a-istiod-ca-cert" pod "327437b8-3b27-4490-9cf6-d965799a1c3a" (UID: "327437b8-3b27-4490-9cf6-d965799a1c3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:48.220476 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.220114 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-data" pod "acfc1375-2b70-4aec-b161-4e24c39a8d59" (UID: "acfc1375-2b70-4aec-b161-4e24c39a8d59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.220788 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.220758 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-podinfo" pod "327437b8-3b27-4490-9cf6-d965799a1c3a" (UID: "327437b8-3b27-4490-9cf6-d965799a1c3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:46:48.222276 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.222225 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-envoy" pod "327437b8-3b27-4490-9cf6-d965799a1c3a" (UID: "327437b8-3b27-4490-9cf6-d965799a1c3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.223146 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.223112 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acfc1375-2b70-4aec-b161-4e24c39a8d59-kube-api-access-87cv5" pod "acfc1375-2b70-4aec-b161-4e24c39a8d59" (UID: "acfc1375-2b70-4aec-b161-4e24c39a8d59"). InnerVolumeSpecName "kube-api-access-87cv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:48.223241 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.223124 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-envoy" pod "acfc1375-2b70-4aec-b161-4e24c39a8d59" (UID: "acfc1375-2b70-4aec-b161-4e24c39a8d59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:48.223407 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.223368 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-token" pod "acfc1375-2b70-4aec-b161-4e24c39a8d59" (UID: "acfc1375-2b70-4aec-b161-4e24c39a8d59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:48.223531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.223467 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327437b8-3b27-4490-9cf6-d965799a1c3a-kube-api-access-d78dz" pod "327437b8-3b27-4490-9cf6-d965799a1c3a" (UID: "327437b8-3b27-4490-9cf6-d965799a1c3a"). InnerVolumeSpecName "kube-api-access-d78dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:48.223797 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.223768 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-podinfo" pod "acfc1375-2b70-4aec-b161-4e24c39a8d59" (UID: "acfc1375-2b70-4aec-b161-4e24c39a8d59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:46:48.224325 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.224301 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-token" pod "327437b8-3b27-4490-9cf6-d965799a1c3a" (UID: "327437b8-3b27-4490-9cf6-d965799a1c3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:48.226165 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.226144 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v"] Nov 03 15:46:48.289742 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.289657 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4"] Nov 03 15:46:48.320925 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.320894 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.320925 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.320923 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.321116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.320936 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.321116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.320944 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/327437b8-3b27-4490-9cf6-d965799a1c3a-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.321116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.320953 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.321116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.320961 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.321116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.320971 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d78dz\" (UniqueName: \"kubernetes.io/projected/327437b8-3b27-4490-9cf6-d965799a1c3a-kube-api-access-d78dz\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.321116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.320980 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87cv5\" (UniqueName: \"kubernetes.io/projected/acfc1375-2b70-4aec-b161-4e24c39a8d59-kube-api-access-87cv5\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.321116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.320990 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.321116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.320998 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acfc1375-2b70-4aec-b161-4e24c39a8d59-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.321116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.321007 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.321116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.321015 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/327437b8-3b27-4490-9cf6-d965799a1c3a-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:48.469765 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.469727 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:48.917549 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.917522 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" Nov 03 15:46:48.917549 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.917538 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd" event={"ID":"327437b8-3b27-4490-9cf6-d965799a1c3a","Type":"ContainerDied","Data":"b47a85b3a7ebb294d769bb4df22f2c8758d07ba2b19204c25fcf5a2b6f9d7b67"} Nov 03 15:46:48.917812 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.917648 4557 scope.go:122] "RemoveContainer" containerID="6b7f1e2478ece1d7bc4c704c57719f10fc9f762c68a735aa04c0b1e7e7f5606d" Nov 03 15:46:48.919054 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.919033 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" Nov 03 15:46:48.919181 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.919057 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb" event={"ID":"acfc1375-2b70-4aec-b161-4e24c39a8d59","Type":"ContainerDied","Data":"16e299be3d2ca7a71d4e20ec264c0531456681ade8ac49edb6cc63a85298931b"} Nov 03 15:46:48.920513 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.920494 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" event={"ID":"2a6c2b03-8f2a-47cd-8e66-923505d511d7","Type":"ContainerDied","Data":"0329d7eb4e67010e91205d6e476a346034a9233398ebc3cc1897104e263a0de1"} Nov 03 15:46:48.920628 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.920536 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp" Nov 03 15:46:48.927612 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.927570 4557 scope.go:122] "RemoveContainer" containerID="15cc961e3e3f64b185896db9d129d4d27f921b1c188aa480feaa7ec359b0ac2c" Nov 03 15:46:48.936477 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:48.936457 4557 scope.go:122] "RemoveContainer" containerID="c0eccdc48adfeedf19b06d2e30b748a2704e88b8c55d8b15df46fdf90dcf3c9d" Nov 03 15:46:49.126225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:49.126189 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp"] Nov 03 15:46:49.176183 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:49.176103 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-qzwmp"] Nov 03 15:46:49.281350 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:49.281318 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd"] Nov 03 15:46:49.289383 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:49.289352 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-kgxgd"] Nov 03 15:46:49.405752 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:49.405718 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb"] Nov 03 15:46:49.412631 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:49.412604 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-mslrb"] Nov 03 15:46:50.285566 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.285531 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a6c2b03-8f2a-47cd-8e66-923505d511d7" path="/var/lib/kubelet/pods/2a6c2b03-8f2a-47cd-8e66-923505d511d7/volumes" Nov 03 15:46:50.286041 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.286026 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="327437b8-3b27-4490-9cf6-d965799a1c3a" path="/var/lib/kubelet/pods/327437b8-3b27-4490-9cf6-d965799a1c3a/volumes" Nov 03 15:46:50.286437 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.286425 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acfc1375-2b70-4aec-b161-4e24c39a8d59" path="/var/lib/kubelet/pods/acfc1375-2b70-4aec-b161-4e24c39a8d59/volumes" Nov 03 15:46:50.395618 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.395574 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb"] Nov 03 15:46:50.396011 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.395997 4557 cpu_manager.go:414] "containerMap: removing container" podUID="acfc1375-2b70-4aec-b161-4e24c39a8d59" containerName="istio-proxy" Nov 03 15:46:50.396067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.396012 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acfc1375-2b70-4aec-b161-4e24c39a8d59" containerName="istio-proxy" Nov 03 15:46:50.396067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.396039 4557 cpu_manager.go:414] "containerMap: removing container" podUID="327437b8-3b27-4490-9cf6-d965799a1c3a" containerName="istio-proxy" Nov 03 15:46:50.396067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.396045 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="327437b8-3b27-4490-9cf6-d965799a1c3a" containerName="istio-proxy" Nov 03 15:46:50.396067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.396060 4557 cpu_manager.go:414] "containerMap: removing container" podUID="2ac71731-3b40-418b-8ddf-816a8aafd5e7" containerName="revoke-keys" Nov 03 15:46:50.396067 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.396065 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ac71731-3b40-418b-8ddf-816a8aafd5e7" containerName="revoke-keys" Nov 03 15:46:50.396230 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.396074 4557 cpu_manager.go:414] "containerMap: removing container" podUID="2a6c2b03-8f2a-47cd-8e66-923505d511d7" containerName="istio-proxy" Nov 03 15:46:50.396230 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.396078 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a6c2b03-8f2a-47cd-8e66-923505d511d7" containerName="istio-proxy" Nov 03 15:46:50.396230 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.396138 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ac71731-3b40-418b-8ddf-816a8aafd5e7" containerName="revoke-keys" Nov 03 15:46:50.396230 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.396146 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a6c2b03-8f2a-47cd-8e66-923505d511d7" containerName="istio-proxy" Nov 03 15:46:50.396230 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.396153 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="acfc1375-2b70-4aec-b161-4e24c39a8d59" containerName="istio-proxy" Nov 03 15:46:50.396230 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.396161 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="3503bbfc-0896-44d5-8b21-02bfa7a360d0" containerName="istio-proxy" Nov 03 15:46:50.396230 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.396170 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="327437b8-3b27-4490-9cf6-d965799a1c3a" containerName="istio-proxy" Nov 03 15:46:50.401154 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.401131 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb" Nov 03 15:46:50.462104 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.462082 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:46:50.478968 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.478945 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:46:50.514918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.514888 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb"] Nov 03 15:46:50.543717 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.543624 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqgxl\" (UniqueName: \"kubernetes.io/projected/0b96a745-452f-4a97-b177-abe512337570-kube-api-access-rqgxl\") pod \"maas-api-revoke-sub-cb2511a3-hbdfb\" (UID: \"0b96a745-452f-4a97-b177-abe512337570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb" Nov 03 15:46:50.543850 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.543725 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b96a745-452f-4a97-b177-abe512337570-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb2511a3-hbdfb\" (UID: \"0b96a745-452f-4a97-b177-abe512337570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb" Nov 03 15:46:50.543920 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.543904 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b96a745-452f-4a97-b177-abe512337570-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb2511a3-hbdfb\" (UID: \"0b96a745-452f-4a97-b177-abe512337570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb" Nov 03 15:46:50.551161 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.551133 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7"] Nov 03 15:46:50.645335 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.645297 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b96a745-452f-4a97-b177-abe512337570-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb2511a3-hbdfb\" (UID: \"0b96a745-452f-4a97-b177-abe512337570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb" Nov 03 15:46:50.645516 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.645354 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqgxl\" (UniqueName: \"kubernetes.io/projected/0b96a745-452f-4a97-b177-abe512337570-kube-api-access-rqgxl\") pod \"maas-api-revoke-sub-cb2511a3-hbdfb\" (UID: \"0b96a745-452f-4a97-b177-abe512337570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb" Nov 03 15:46:50.645516 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.645378 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b96a745-452f-4a97-b177-abe512337570-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb2511a3-hbdfb\" (UID: \"0b96a745-452f-4a97-b177-abe512337570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb" Nov 03 15:46:50.646060 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.646030 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b96a745-452f-4a97-b177-abe512337570-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb2511a3-hbdfb\" (UID: \"0b96a745-452f-4a97-b177-abe512337570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb" Nov 03 15:46:50.647699 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.647679 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b96a745-452f-4a97-b177-abe512337570-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb2511a3-hbdfb\" (UID: \"0b96a745-452f-4a97-b177-abe512337570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb" Nov 03 15:46:50.716235 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.716201 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqgxl\" (UniqueName: \"kubernetes.io/projected/0b96a745-452f-4a97-b177-abe512337570-kube-api-access-rqgxl\") pod \"maas-api-revoke-sub-cb2511a3-hbdfb\" (UID: \"0b96a745-452f-4a97-b177-abe512337570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb" Nov 03 15:46:50.720102 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.720078 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6"] Nov 03 15:46:50.724785 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.724765 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6" Nov 03 15:46:50.799703 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.799625 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6"] Nov 03 15:46:50.847790 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.847760 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg8s8\" (UniqueName: \"kubernetes.io/projected/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-kube-api-access-pg8s8\") pod \"maas-api-revoke-sub-cab9a0da-j27d6\" (UID: \"b16ddb97-4e08-46ba-bf10-8a24dbd02b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6" Nov 03 15:46:50.847970 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.847893 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cab9a0da-j27d6\" (UID: \"b16ddb97-4e08-46ba-bf10-8a24dbd02b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6" Nov 03 15:46:50.847970 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.847958 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cab9a0da-j27d6\" (UID: \"b16ddb97-4e08-46ba-bf10-8a24dbd02b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6" Nov 03 15:46:50.948657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.948606 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cab9a0da-j27d6\" (UID: \"b16ddb97-4e08-46ba-bf10-8a24dbd02b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6" Nov 03 15:46:50.948859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.948700 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cab9a0da-j27d6\" (UID: \"b16ddb97-4e08-46ba-bf10-8a24dbd02b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6" Nov 03 15:46:50.948859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.948805 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg8s8\" (UniqueName: \"kubernetes.io/projected/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-kube-api-access-pg8s8\") pod \"maas-api-revoke-sub-cab9a0da-j27d6\" (UID: \"b16ddb97-4e08-46ba-bf10-8a24dbd02b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6" Nov 03 15:46:50.949416 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.949390 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cab9a0da-j27d6\" (UID: \"b16ddb97-4e08-46ba-bf10-8a24dbd02b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6" Nov 03 15:46:50.951115 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:50.951086 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cab9a0da-j27d6\" (UID: \"b16ddb97-4e08-46ba-bf10-8a24dbd02b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6" Nov 03 15:46:51.017374 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:51.017301 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb" Nov 03 15:46:51.049243 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:51.049208 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:51.054982 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:51.054909 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg8s8\" (UniqueName: \"kubernetes.io/projected/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-kube-api-access-pg8s8\") pod \"maas-api-revoke-sub-cab9a0da-j27d6\" (UID: \"b16ddb97-4e08-46ba-bf10-8a24dbd02b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6" Nov 03 15:46:51.245413 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:51.245377 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:51.339575 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:51.339489 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6" Nov 03 15:46:51.411620 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:51.410533 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb"] Nov 03 15:46:51.416749 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:46:51.416723 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b96a745_452f_4a97_b177_abe512337570.slice/crio-13b596af03dbf642efbefb43b372e92e44b78e24a929f9d7f01e51461bae5c14 WatchSource:0}: Error finding container 13b596af03dbf642efbefb43b372e92e44b78e24a929f9d7f01e51461bae5c14: Status 404 returned error can't find the container with id 13b596af03dbf642efbefb43b372e92e44b78e24a929f9d7f01e51461bae5c14 Nov 03 15:46:51.733840 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:51.733806 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6"] Nov 03 15:46:51.735143 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:46:51.735114 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16ddb97_4e08_46ba_bf10_8a24dbd02b26.slice/crio-d6067a3450a579ecfc82daab1263a72dbc1fe07b330d0017411424fcd3cf4a3a WatchSource:0}: Error finding container d6067a3450a579ecfc82daab1263a72dbc1fe07b330d0017411424fcd3cf4a3a: Status 404 returned error can't find the container with id d6067a3450a579ecfc82daab1263a72dbc1fe07b330d0017411424fcd3cf4a3a Nov 03 15:46:51.946838 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:51.946750 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6" event={"ID":"b16ddb97-4e08-46ba-bf10-8a24dbd02b26","Type":"ContainerStarted","Data":"d6067a3450a579ecfc82daab1263a72dbc1fe07b330d0017411424fcd3cf4a3a"} Nov 03 15:46:51.948459 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:51.948429 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb" event={"ID":"0b96a745-452f-4a97-b177-abe512337570","Type":"ContainerStarted","Data":"4e032cb8127b1e9c5f0bd9ed07c45ad08d245ce07fe27a858c59840363675727"} Nov 03 15:46:51.948617 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:51.948490 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb" event={"ID":"0b96a745-452f-4a97-b177-abe512337570","Type":"ContainerStarted","Data":"13b596af03dbf642efbefb43b372e92e44b78e24a929f9d7f01e51461bae5c14"} Nov 03 15:46:52.133200 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:52.133148 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb" podStartSLOduration=2.133133683 podStartE2EDuration="2.133133683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:46:52.131241235 +0000 UTC m=+2321.974362780" watchObservedRunningTime="2026-11-03 15:46:52.133133683 +0000 UTC m=+2321.976255227" Nov 03 15:46:52.913228 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:52.913195 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:52.956465 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:52.956428 4557 generic.go:356] "Generic (PLEG): container finished" podID="b16ddb97-4e08-46ba-bf10-8a24dbd02b26" containerID="a38690967e9f9ff121f9254ee4f17b404658f9bec5ca5d31be4e1e4f5e337f90" exitCode=0 Nov 03 15:46:52.956654 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:52.956520 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6" event={"ID":"b16ddb97-4e08-46ba-bf10-8a24dbd02b26","Type":"ContainerDied","Data":"a38690967e9f9ff121f9254ee4f17b404658f9bec5ca5d31be4e1e4f5e337f90"} Nov 03 15:46:52.958214 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:52.958190 4557 generic.go:356] "Generic (PLEG): container finished" podID="0b96a745-452f-4a97-b177-abe512337570" containerID="4e032cb8127b1e9c5f0bd9ed07c45ad08d245ce07fe27a858c59840363675727" exitCode=0 Nov 03 15:46:52.958338 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:52.958225 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb" event={"ID":"0b96a745-452f-4a97-b177-abe512337570","Type":"ContainerDied","Data":"4e032cb8127b1e9c5f0bd9ed07c45ad08d245ce07fe27a858c59840363675727"} Nov 03 15:46:53.252511 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:53.252475 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:53.455168 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:53.455137 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq"] Nov 03 15:46:53.457504 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:53.457469 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9855ff-v58wq"] Nov 03 15:46:53.472054 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:53.472023 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:53.669507 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:53.669476 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:53.940098 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:53.939996 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:54.150103 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.149569 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb" Nov 03 15:46:54.152857 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.152838 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6" Nov 03 15:46:54.220147 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.220071 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-maas-api-openshift-ca\") pod \"b16ddb97-4e08-46ba-bf10-8a24dbd02b26\" (UID: \"b16ddb97-4e08-46ba-bf10-8a24dbd02b26\") " Nov 03 15:46:54.220298 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.220189 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b96a745-452f-4a97-b177-abe512337570-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b96a745-452f-4a97-b177-abe512337570-maas-api-xks-ca\") pod \"0b96a745-452f-4a97-b177-abe512337570\" (UID: \"0b96a745-452f-4a97-b177-abe512337570\") " Nov 03 15:46:54.220298 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.220235 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b96a745-452f-4a97-b177-abe512337570-kube-api-access-rqgxl\" (UniqueName: \"kubernetes.io/projected/0b96a745-452f-4a97-b177-abe512337570-kube-api-access-rqgxl\") pod \"0b96a745-452f-4a97-b177-abe512337570\" (UID: \"0b96a745-452f-4a97-b177-abe512337570\") " Nov 03 15:46:54.220435 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.220326 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b96a745-452f-4a97-b177-abe512337570-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b96a745-452f-4a97-b177-abe512337570-maas-api-openshift-ca\") pod \"0b96a745-452f-4a97-b177-abe512337570\" (UID: \"0b96a745-452f-4a97-b177-abe512337570\") " Nov 03 15:46:54.220435 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.220357 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-kube-api-access-pg8s8\" (UniqueName: \"kubernetes.io/projected/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-kube-api-access-pg8s8\") pod \"b16ddb97-4e08-46ba-bf10-8a24dbd02b26\" (UID: \"b16ddb97-4e08-46ba-bf10-8a24dbd02b26\") " Nov 03 15:46:54.220435 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.220392 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-maas-api-xks-ca\") pod \"b16ddb97-4e08-46ba-bf10-8a24dbd02b26\" (UID: \"b16ddb97-4e08-46ba-bf10-8a24dbd02b26\") " Nov 03 15:46:54.220623 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.220519 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-maas-api-openshift-ca" pod "b16ddb97-4e08-46ba-bf10-8a24dbd02b26" (UID: "b16ddb97-4e08-46ba-bf10-8a24dbd02b26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:54.220832 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.220787 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b96a745-452f-4a97-b177-abe512337570-maas-api-openshift-ca" pod "0b96a745-452f-4a97-b177-abe512337570" (UID: "0b96a745-452f-4a97-b177-abe512337570"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:46:54.220994 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.220969 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:54.221087 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.220998 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b96a745-452f-4a97-b177-abe512337570-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:54.222483 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.222449 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b96a745-452f-4a97-b177-abe512337570-maas-api-xks-ca" pod "0b96a745-452f-4a97-b177-abe512337570" (UID: "0b96a745-452f-4a97-b177-abe512337570"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:54.222666 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.222570 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b96a745-452f-4a97-b177-abe512337570-kube-api-access-rqgxl" pod "0b96a745-452f-4a97-b177-abe512337570" (UID: "0b96a745-452f-4a97-b177-abe512337570"). InnerVolumeSpecName "kube-api-access-rqgxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:54.222762 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.222738 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-kube-api-access-pg8s8" pod "b16ddb97-4e08-46ba-bf10-8a24dbd02b26" (UID: "b16ddb97-4e08-46ba-bf10-8a24dbd02b26"). InnerVolumeSpecName "kube-api-access-pg8s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:54.222869 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.222852 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-maas-api-xks-ca" pod "b16ddb97-4e08-46ba-bf10-8a24dbd02b26" (UID: "b16ddb97-4e08-46ba-bf10-8a24dbd02b26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:54.295101 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.295065 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a3395af-d9ae-443c-a43e-54aeacc17c81" path="/var/lib/kubelet/pods/6a3395af-d9ae-443c-a43e-54aeacc17c81/volumes" Nov 03 15:46:54.321701 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.321675 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b96a745-452f-4a97-b177-abe512337570-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:54.321701 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.321699 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqgxl\" (UniqueName: \"kubernetes.io/projected/0b96a745-452f-4a97-b177-abe512337570-kube-api-access-rqgxl\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:54.321851 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.321710 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg8s8\" (UniqueName: \"kubernetes.io/projected/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-kube-api-access-pg8s8\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:54.321851 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.321719 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b16ddb97-4e08-46ba-bf10-8a24dbd02b26-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:54.972204 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.972167 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6" event={"ID":"b16ddb97-4e08-46ba-bf10-8a24dbd02b26","Type":"ContainerDied","Data":"d6067a3450a579ecfc82daab1263a72dbc1fe07b330d0017411424fcd3cf4a3a"} Nov 03 15:46:54.972747 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.972263 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6067a3450a579ecfc82daab1263a72dbc1fe07b330d0017411424fcd3cf4a3a" Nov 03 15:46:54.972747 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.972178 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6" Nov 03 15:46:54.973624 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.973599 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb" event={"ID":"0b96a745-452f-4a97-b177-abe512337570","Type":"ContainerDied","Data":"13b596af03dbf642efbefb43b372e92e44b78e24a929f9d7f01e51461bae5c14"} Nov 03 15:46:54.973624 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.973621 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb" Nov 03 15:46:54.973794 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:54.973636 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13b596af03dbf642efbefb43b372e92e44b78e24a929f9d7f01e51461bae5c14" Nov 03 15:46:55.551635 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:55.551592 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" podUID="cf100201-cb1e-462e-878e-ff94690746d7" containerName="payload-processing" containerID="cri-o://5f1886533aae05aaf35fe19bbc428c70c2101e18c9cf3a143f5e96c86f6cdf46" gracePeriod=25 Nov 03 15:46:55.742913 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:55.742857 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:55.942350 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:55.942264 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:58.322478 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.322432 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" containerID="cri-o://7af0ef93537d29708a9a63466c2daacb69de36ae93fef8221e14fbe0701dfb93" gracePeriod=45 Nov 03 15:46:58.340412 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.340375 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" containerID="cri-o://3cc1bfcf2badb40cae3e7708ac52d0c830e69abf2f871166f8dc4856ab85c4eb" gracePeriod=45 Nov 03 15:46:58.611405 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.611382 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:46:58.614494 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.614470 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:46:58.786531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.786500 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-model-cache\") pod \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " Nov 03 15:46:58.786531 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.786542 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-tmp-dir\") pod \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " Nov 03 15:46:58.786807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.786607 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-tls-certs\" (UniqueName: \"kubernetes.io/secret/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-tls-certs\") pod \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " Nov 03 15:46:58.786807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.786656 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-home\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-home\") pod \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " Nov 03 15:46:58.786807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.786736 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-home\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-home\") pod \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " Nov 03 15:46:58.786807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.786760 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-dshm\") pod \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " Nov 03 15:46:58.786807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.786792 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-tmp-dir\") pod \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " Nov 03 15:46:58.787075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.786833 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-dshm\") pod \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " Nov 03 15:46:58.787075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.786841 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-model-cache" pod "f8216cdd-2b2d-43ee-b557-6a14c8152b3f" (UID: "f8216cdd-2b2d-43ee-b557-6a14c8152b3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:58.787075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.786852 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-tmp-dir" pod "f8216cdd-2b2d-43ee-b557-6a14c8152b3f" (UID: "f8216cdd-2b2d-43ee-b557-6a14c8152b3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:58.787075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.786868 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-kube-api-access-d5jn5\" (UniqueName: \"kubernetes.io/projected/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-kube-api-access-d5jn5\") pod \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\" (UID: \"f8216cdd-2b2d-43ee-b557-6a14c8152b3f\") " Nov 03 15:46:58.787075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.786925 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-tls-certs\") pod \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " Nov 03 15:46:58.787075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.786958 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-model-cache\") pod \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " Nov 03 15:46:58.787451 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.787088 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-tmp-dir" pod "0ea2141e-d478-4cf9-b465-cc75a7bc6dec" (UID: "0ea2141e-d478-4cf9-b465-cc75a7bc6dec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:58.787451 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.787292 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-home" pod "0ea2141e-d478-4cf9-b465-cc75a7bc6dec" (UID: "0ea2141e-d478-4cf9-b465-cc75a7bc6dec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:58.787574 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.787517 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-home" pod "f8216cdd-2b2d-43ee-b557-6a14c8152b3f" (UID: "f8216cdd-2b2d-43ee-b557-6a14c8152b3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:58.787657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.786993 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-kube-api-access-2ztn6\" (UniqueName: \"kubernetes.io/projected/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-kube-api-access-2ztn6\") pod \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\" (UID: \"0ea2141e-d478-4cf9-b465-cc75a7bc6dec\") " Nov 03 15:46:58.787755 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.787734 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-model-cache" pod "0ea2141e-d478-4cf9-b465-cc75a7bc6dec" (UID: "0ea2141e-d478-4cf9-b465-cc75a7bc6dec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:58.788015 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.787996 4557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-home\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:58.788084 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.788019 4557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-home\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:58.788084 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.788031 4557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-tmp-dir\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:58.788084 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.788046 4557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-model-cache\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:58.788084 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.788060 4557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-model-cache\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:58.788084 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.788073 4557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-tmp-dir\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:58.790788 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.790329 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-kube-api-access-2ztn6" pod "0ea2141e-d478-4cf9-b465-cc75a7bc6dec" (UID: "0ea2141e-d478-4cf9-b465-cc75a7bc6dec"). InnerVolumeSpecName "kube-api-access-2ztn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:58.791341 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.791298 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-tls-certs" pod "0ea2141e-d478-4cf9-b465-cc75a7bc6dec" (UID: "0ea2141e-d478-4cf9-b465-cc75a7bc6dec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:58.797025 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.797005 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-dshm" pod "0ea2141e-d478-4cf9-b465-cc75a7bc6dec" (UID: "0ea2141e-d478-4cf9-b465-cc75a7bc6dec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:58.797165 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.797033 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-dshm" pod "f8216cdd-2b2d-43ee-b557-6a14c8152b3f" (UID: "f8216cdd-2b2d-43ee-b557-6a14c8152b3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:46:58.797247 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.797165 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-tls-certs" pod "f8216cdd-2b2d-43ee-b557-6a14c8152b3f" (UID: "f8216cdd-2b2d-43ee-b557-6a14c8152b3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:46:58.798817 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.798796 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-kube-api-access-d5jn5" pod "f8216cdd-2b2d-43ee-b557-6a14c8152b3f" (UID: "f8216cdd-2b2d-43ee-b557-6a14c8152b3f"). InnerVolumeSpecName "kube-api-access-d5jn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:46:58.890206 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.890109 4557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-dshm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:58.890206 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.890146 4557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-dshm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:58.890206 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.890159 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5jn5\" (UniqueName: \"kubernetes.io/projected/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-kube-api-access-d5jn5\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:58.890206 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.890172 4557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-tls-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:58.890206 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.890185 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ztn6\" (UniqueName: \"kubernetes.io/projected/0ea2141e-d478-4cf9-b465-cc75a7bc6dec-kube-api-access-2ztn6\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:58.890206 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.890201 4557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8216cdd-2b2d-43ee-b557-6a14c8152b3f-tls-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:46:58.958410 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.958375 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:46:58.999322 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.999290 4557 generic.go:356] "Generic (PLEG): container finished" podID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerID="3cc1bfcf2badb40cae3e7708ac52d0c830e69abf2f871166f8dc4856ab85c4eb" exitCode=0 Nov 03 15:46:58.999541 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.999340 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" Nov 03 15:46:58.999541 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.999339 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" event={"ID":"f8216cdd-2b2d-43ee-b557-6a14c8152b3f","Type":"ContainerDied","Data":"3cc1bfcf2badb40cae3e7708ac52d0c830e69abf2f871166f8dc4856ab85c4eb"} Nov 03 15:46:58.999541 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.999394 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs" event={"ID":"f8216cdd-2b2d-43ee-b557-6a14c8152b3f","Type":"ContainerDied","Data":"f350568b90f8d3edbd8e51e60674aedaf90a2ffecf45bf1c6ed705a8789e2dc9"} Nov 03 15:46:58.999541 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:58.999423 4557 scope.go:122] "RemoveContainer" containerID="3cc1bfcf2badb40cae3e7708ac52d0c830e69abf2f871166f8dc4856ab85c4eb" Nov 03 15:46:59.001144 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:59.001116 4557 generic.go:356] "Generic (PLEG): container finished" podID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerID="7af0ef93537d29708a9a63466c2daacb69de36ae93fef8221e14fbe0701dfb93" exitCode=0 Nov 03 15:46:59.001254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:59.001152 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" Nov 03 15:46:59.001254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:59.001173 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" event={"ID":"0ea2141e-d478-4cf9-b465-cc75a7bc6dec","Type":"ContainerDied","Data":"7af0ef93537d29708a9a63466c2daacb69de36ae93fef8221e14fbe0701dfb93"} Nov 03 15:46:59.001254 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:59.001204 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7" event={"ID":"0ea2141e-d478-4cf9-b465-cc75a7bc6dec","Type":"ContainerDied","Data":"c9420314b83f9f044228ae4c5eaa8c9c39a83157b7610917c0a8b875733d4109"} Nov 03 15:46:59.010136 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:59.010115 4557 scope.go:122] "RemoveContainer" containerID="3cc1bfcf2badb40cae3e7708ac52d0c830e69abf2f871166f8dc4856ab85c4eb" Nov 03 15:46:59.010450 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:46:59.010428 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc1bfcf2badb40cae3e7708ac52d0c830e69abf2f871166f8dc4856ab85c4eb\": container with ID starting with 3cc1bfcf2badb40cae3e7708ac52d0c830e69abf2f871166f8dc4856ab85c4eb not found: ID does not exist" containerID="3cc1bfcf2badb40cae3e7708ac52d0c830e69abf2f871166f8dc4856ab85c4eb" Nov 03 15:46:59.010567 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:59.010457 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc1bfcf2badb40cae3e7708ac52d0c830e69abf2f871166f8dc4856ab85c4eb"} err="failed to get container status \"3cc1bfcf2badb40cae3e7708ac52d0c830e69abf2f871166f8dc4856ab85c4eb\": rpc error: code = NotFound desc = could not find container \"3cc1bfcf2badb40cae3e7708ac52d0c830e69abf2f871166f8dc4856ab85c4eb\": container with ID starting with 3cc1bfcf2badb40cae3e7708ac52d0c830e69abf2f871166f8dc4856ab85c4eb not found: ID does not exist" Nov 03 15:46:59.010567 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:59.010477 4557 scope.go:122] "RemoveContainer" containerID="7af0ef93537d29708a9a63466c2daacb69de36ae93fef8221e14fbe0701dfb93" Nov 03 15:46:59.019535 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:59.019515 4557 scope.go:122] "RemoveContainer" containerID="7af0ef93537d29708a9a63466c2daacb69de36ae93fef8221e14fbe0701dfb93" Nov 03 15:46:59.019858 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:46:59.019837 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af0ef93537d29708a9a63466c2daacb69de36ae93fef8221e14fbe0701dfb93\": container with ID starting with 7af0ef93537d29708a9a63466c2daacb69de36ae93fef8221e14fbe0701dfb93 not found: ID does not exist" containerID="7af0ef93537d29708a9a63466c2daacb69de36ae93fef8221e14fbe0701dfb93" Nov 03 15:46:59.019930 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:59.019865 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af0ef93537d29708a9a63466c2daacb69de36ae93fef8221e14fbe0701dfb93"} err="failed to get container status \"7af0ef93537d29708a9a63466c2daacb69de36ae93fef8221e14fbe0701dfb93\": rpc error: code = NotFound desc = could not find container \"7af0ef93537d29708a9a63466c2daacb69de36ae93fef8221e14fbe0701dfb93\": container with ID starting with 7af0ef93537d29708a9a63466c2daacb69de36ae93fef8221e14fbe0701dfb93 not found: ID does not exist" Nov 03 15:46:59.187542 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:59.187456 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7"] Nov 03 15:46:59.198117 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:59.198090 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/premium-w0-e62ff3-kserve-5bf657f878-wmtb7"] Nov 03 15:46:59.312249 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:59.312218 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs"] Nov 03 15:46:59.321646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:46:59.321619 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/simulator-w0-e62ff3-kserve-74dd5cbd59-dvgjs"] Nov 03 15:47:00.292276 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:00.292236 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" path="/var/lib/kubelet/pods/0ea2141e-d478-4cf9-b465-cc75a7bc6dec/volumes" Nov 03 15:47:00.294869 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:00.292812 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" path="/var/lib/kubelet/pods/f8216cdd-2b2d-43ee-b557-6a14c8152b3f/volumes" Nov 03 15:47:01.157014 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:01.156983 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:01.354357 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:01.354325 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:01.537973 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:01.537886 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:01.929926 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:01.929883 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:02.146536 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:02.146493 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:02.297878 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:02.297846 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg"] Nov 03 15:47:02.298061 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:02.298037 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" podUID="b37f9ec1-b784-41fd-9a79-8df424aa77e5" containerName="istio-proxy" containerID="cri-o://b7c46f81e707c8a068eeac7090cbca8557434d8e002d0dab988d451bd5ef8c7d" gracePeriod=30 Nov 03 15:47:02.388606 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:02.385976 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:03.246376 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.246260 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" containerID="cri-o://8cc513c36f8d563661286f7d6492344d3dfa63e16ced3ab23979e9107ffe1500" gracePeriod=45 Nov 03 15:47:03.323247 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.323209 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" containerID="cri-o://60eeb37eb73bd7b6985faad3d28f43ebb6f134f942a68c1e2154943c4fc04308" gracePeriod=45 Nov 03 15:47:03.597003 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.596977 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:47:03.600132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.600116 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:47:03.640708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.640683 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-tmp-dir\") pod \"9c968070-2d65-4501-b504-e96664a97f8d\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " Nov 03 15:47:03.640866 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.640734 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-home\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-home\") pod \"9c968070-2d65-4501-b504-e96664a97f8d\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " Nov 03 15:47:03.640866 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.640756 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-model-cache\") pod \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " Nov 03 15:47:03.640866 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.640774 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-home\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-home\") pod \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " Nov 03 15:47:03.640866 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.640797 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-tmp-dir\") pod \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " Nov 03 15:47:03.640866 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.640832 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-model-cache\") pod \"9c968070-2d65-4501-b504-e96664a97f8d\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " Nov 03 15:47:03.640866 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.640854 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-dshm\") pod \"9c968070-2d65-4501-b504-e96664a97f8d\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " Nov 03 15:47:03.641189 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.640905 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c968070-2d65-4501-b504-e96664a97f8d-kube-api-access-ggrpb\" (UniqueName: \"kubernetes.io/projected/9c968070-2d65-4501-b504-e96664a97f8d-kube-api-access-ggrpb\") pod \"9c968070-2d65-4501-b504-e96664a97f8d\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " Nov 03 15:47:03.641189 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.640906 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-tmp-dir" pod "9c968070-2d65-4501-b504-e96664a97f8d" (UID: "9c968070-2d65-4501-b504-e96664a97f8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:03.641189 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.641039 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-tls-certs\") pod \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " Nov 03 15:47:03.641189 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.641051 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-home" pod "9c968070-2d65-4501-b504-e96664a97f8d" (UID: "9c968070-2d65-4501-b504-e96664a97f8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:03.641189 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.641066 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-home" pod "bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" (UID: "bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:03.641189 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.641057 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-model-cache" pod "bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" (UID: "bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:03.641189 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.641089 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c968070-2d65-4501-b504-e96664a97f8d-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c968070-2d65-4501-b504-e96664a97f8d-tls-certs\") pod \"9c968070-2d65-4501-b504-e96664a97f8d\" (UID: \"9c968070-2d65-4501-b504-e96664a97f8d\") " Nov 03 15:47:03.641189 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.641109 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-model-cache" pod "9c968070-2d65-4501-b504-e96664a97f8d" (UID: "9c968070-2d65-4501-b504-e96664a97f8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:03.641189 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.641133 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-kube-api-access-7dhds\" (UniqueName: \"kubernetes.io/projected/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-kube-api-access-7dhds\") pod \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " Nov 03 15:47:03.641189 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.641168 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-dshm\") pod \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\" (UID: \"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc\") " Nov 03 15:47:03.641771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.641367 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-tmp-dir" pod "bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" (UID: "bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:03.641771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.641627 4557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-home\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:03.641771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.641646 4557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-model-cache\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:03.641771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.641662 4557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-home\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:03.641771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.641677 4557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-tmp-dir\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:03.641771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.641691 4557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-model-cache\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:03.641771 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.641705 4557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-tmp-dir\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:03.643511 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.643478 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-dshm" pod "bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" (UID: "bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:03.643511 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.643498 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-kube-api-access-7dhds" pod "bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" (UID: "bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc"). InnerVolumeSpecName "kube-api-access-7dhds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:03.643682 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.643564 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c968070-2d65-4501-b504-e96664a97f8d-kube-api-access-ggrpb" pod "9c968070-2d65-4501-b504-e96664a97f8d" (UID: "9c968070-2d65-4501-b504-e96664a97f8d"). InnerVolumeSpecName "kube-api-access-ggrpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:03.643682 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.643569 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c968070-2d65-4501-b504-e96664a97f8d-tls-certs" pod "9c968070-2d65-4501-b504-e96664a97f8d" (UID: "9c968070-2d65-4501-b504-e96664a97f8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:47:03.643682 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.643634 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-tls-certs" pod "bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" (UID: "bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:47:03.643682 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.643660 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-dshm" pod "9c968070-2d65-4501-b504-e96664a97f8d" (UID: "9c968070-2d65-4501-b504-e96664a97f8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:03.743217 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.742951 4557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-tls-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:03.743217 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.742989 4557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c968070-2d65-4501-b504-e96664a97f8d-tls-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:03.743217 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.743003 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dhds\" (UniqueName: \"kubernetes.io/projected/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-kube-api-access-7dhds\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:03.743217 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.743017 4557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc-dshm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:03.743217 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.743029 4557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c968070-2d65-4501-b504-e96664a97f8d-dshm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:03.743217 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.743041 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggrpb\" (UniqueName: \"kubernetes.io/projected/9c968070-2d65-4501-b504-e96664a97f8d-kube-api-access-ggrpb\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:03.872457 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.872424 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx"] Nov 03 15:47:03.872864 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.872851 4557 cpu_manager.go:414] "containerMap: removing container" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" Nov 03 15:47:03.872918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.872865 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" Nov 03 15:47:03.872918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.872877 4557 cpu_manager.go:414] "containerMap: removing container" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" Nov 03 15:47:03.872918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.872882 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" Nov 03 15:47:03.872918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.872891 4557 cpu_manager.go:414] "containerMap: removing container" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" Nov 03 15:47:03.872918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.872896 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" Nov 03 15:47:03.872918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.872911 4557 cpu_manager.go:414] "containerMap: removing container" podUID="b16ddb97-4e08-46ba-bf10-8a24dbd02b26" containerName="revoke-keys" Nov 03 15:47:03.873124 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.872916 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16ddb97-4e08-46ba-bf10-8a24dbd02b26" containerName="revoke-keys" Nov 03 15:47:03.873124 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.872942 4557 cpu_manager.go:414] "containerMap: removing container" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" Nov 03 15:47:03.873124 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.872947 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" Nov 03 15:47:03.873124 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.872965 4557 cpu_manager.go:414] "containerMap: removing container" podUID="0b96a745-452f-4a97-b177-abe512337570" containerName="revoke-keys" Nov 03 15:47:03.873124 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.872971 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b96a745-452f-4a97-b177-abe512337570" containerName="revoke-keys" Nov 03 15:47:03.873124 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.873036 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16ddb97-4e08-46ba-bf10-8a24dbd02b26" containerName="revoke-keys" Nov 03 15:47:03.873124 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.873045 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerName="main" Nov 03 15:47:03.873124 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.873054 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b96a745-452f-4a97-b177-abe512337570" containerName="revoke-keys" Nov 03 15:47:03.873124 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.873061 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea2141e-d478-4cf9-b465-cc75a7bc6dec" containerName="main" Nov 03 15:47:03.873124 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.873066 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8216cdd-2b2d-43ee-b557-6a14c8152b3f" containerName="main" Nov 03 15:47:03.873124 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.873071 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c968070-2d65-4501-b504-e96664a97f8d" containerName="main" Nov 03 15:47:03.877641 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.877615 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx" Nov 03 15:47:03.935720 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.935686 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx"] Nov 03 15:47:03.945319 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.945285 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvmlr\" (UniqueName: \"kubernetes.io/projected/b7bd7224-6371-4352-a5ef-37bcf2155cf7-kube-api-access-cvmlr\") pod \"maas-api-revoke-sub-4bd373bc-ktlbx\" (UID: \"b7bd7224-6371-4352-a5ef-37bcf2155cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx" Nov 03 15:47:03.945475 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.945345 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7bd7224-6371-4352-a5ef-37bcf2155cf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bd373bc-ktlbx\" (UID: \"b7bd7224-6371-4352-a5ef-37bcf2155cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx" Nov 03 15:47:03.945524 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.945488 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7bd7224-6371-4352-a5ef-37bcf2155cf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bd373bc-ktlbx\" (UID: \"b7bd7224-6371-4352-a5ef-37bcf2155cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx" Nov 03 15:47:03.978485 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.978440 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:47:03.980754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:03.978519 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:47:04.032687 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.032653 4557 generic.go:356] "Generic (PLEG): container finished" podID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" containerID="60eeb37eb73bd7b6985faad3d28f43ebb6f134f942a68c1e2154943c4fc04308" exitCode=0 Nov 03 15:47:04.032882 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.032702 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" Nov 03 15:47:04.032882 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.032735 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" event={"ID":"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc","Type":"ContainerDied","Data":"60eeb37eb73bd7b6985faad3d28f43ebb6f134f942a68c1e2154943c4fc04308"} Nov 03 15:47:04.032882 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.032781 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4" event={"ID":"bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc","Type":"ContainerDied","Data":"c207583988b9a48162f523b469d9f808ec2b1808a74f8f6b40d180ed92df61f0"} Nov 03 15:47:04.032882 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.032806 4557 scope.go:122] "RemoveContainer" containerID="60eeb37eb73bd7b6985faad3d28f43ebb6f134f942a68c1e2154943c4fc04308" Nov 03 15:47:04.034273 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.034245 4557 generic.go:356] "Generic (PLEG): container finished" podID="9c968070-2d65-4501-b504-e96664a97f8d" containerID="8cc513c36f8d563661286f7d6492344d3dfa63e16ced3ab23979e9107ffe1500" exitCode=0 Nov 03 15:47:04.034382 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.034273 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" event={"ID":"9c968070-2d65-4501-b504-e96664a97f8d","Type":"ContainerDied","Data":"8cc513c36f8d563661286f7d6492344d3dfa63e16ced3ab23979e9107ffe1500"} Nov 03 15:47:04.034382 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.034318 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" event={"ID":"9c968070-2d65-4501-b504-e96664a97f8d","Type":"ContainerDied","Data":"0f4e6609dc4c2706a8e44631433c18beff315e92fe4612c4add8a228941f5945"} Nov 03 15:47:04.034382 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.034334 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v" Nov 03 15:47:04.043678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.043657 4557 scope.go:122] "RemoveContainer" containerID="60eeb37eb73bd7b6985faad3d28f43ebb6f134f942a68c1e2154943c4fc04308" Nov 03 15:47:04.043981 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:47:04.043960 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60eeb37eb73bd7b6985faad3d28f43ebb6f134f942a68c1e2154943c4fc04308\": container with ID starting with 60eeb37eb73bd7b6985faad3d28f43ebb6f134f942a68c1e2154943c4fc04308 not found: ID does not exist" containerID="60eeb37eb73bd7b6985faad3d28f43ebb6f134f942a68c1e2154943c4fc04308" Nov 03 15:47:04.044075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.043989 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60eeb37eb73bd7b6985faad3d28f43ebb6f134f942a68c1e2154943c4fc04308"} err="failed to get container status \"60eeb37eb73bd7b6985faad3d28f43ebb6f134f942a68c1e2154943c4fc04308\": rpc error: code = NotFound desc = could not find container \"60eeb37eb73bd7b6985faad3d28f43ebb6f134f942a68c1e2154943c4fc04308\": container with ID starting with 60eeb37eb73bd7b6985faad3d28f43ebb6f134f942a68c1e2154943c4fc04308 not found: ID does not exist" Nov 03 15:47:04.044075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.044013 4557 scope.go:122] "RemoveContainer" containerID="8cc513c36f8d563661286f7d6492344d3dfa63e16ced3ab23979e9107ffe1500" Nov 03 15:47:04.046640 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.046616 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvmlr\" (UniqueName: \"kubernetes.io/projected/b7bd7224-6371-4352-a5ef-37bcf2155cf7-kube-api-access-cvmlr\") pod \"maas-api-revoke-sub-4bd373bc-ktlbx\" (UID: \"b7bd7224-6371-4352-a5ef-37bcf2155cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx" Nov 03 15:47:04.046736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.046690 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7bd7224-6371-4352-a5ef-37bcf2155cf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bd373bc-ktlbx\" (UID: \"b7bd7224-6371-4352-a5ef-37bcf2155cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx" Nov 03 15:47:04.046736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.046733 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7bd7224-6371-4352-a5ef-37bcf2155cf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bd373bc-ktlbx\" (UID: \"b7bd7224-6371-4352-a5ef-37bcf2155cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx" Nov 03 15:47:04.047412 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.047394 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7bd7224-6371-4352-a5ef-37bcf2155cf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bd373bc-ktlbx\" (UID: \"b7bd7224-6371-4352-a5ef-37bcf2155cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx" Nov 03 15:47:04.049772 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.049753 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7bd7224-6371-4352-a5ef-37bcf2155cf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bd373bc-ktlbx\" (UID: \"b7bd7224-6371-4352-a5ef-37bcf2155cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx" Nov 03 15:47:04.053089 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.053071 4557 scope.go:122] "RemoveContainer" containerID="8cc513c36f8d563661286f7d6492344d3dfa63e16ced3ab23979e9107ffe1500" Nov 03 15:47:04.053369 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:47:04.053351 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc513c36f8d563661286f7d6492344d3dfa63e16ced3ab23979e9107ffe1500\": container with ID starting with 8cc513c36f8d563661286f7d6492344d3dfa63e16ced3ab23979e9107ffe1500 not found: ID does not exist" containerID="8cc513c36f8d563661286f7d6492344d3dfa63e16ced3ab23979e9107ffe1500" Nov 03 15:47:04.053428 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.053375 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc513c36f8d563661286f7d6492344d3dfa63e16ced3ab23979e9107ffe1500"} err="failed to get container status \"8cc513c36f8d563661286f7d6492344d3dfa63e16ced3ab23979e9107ffe1500\": rpc error: code = NotFound desc = could not find container \"8cc513c36f8d563661286f7d6492344d3dfa63e16ced3ab23979e9107ffe1500\": container with ID starting with 8cc513c36f8d563661286f7d6492344d3dfa63e16ced3ab23979e9107ffe1500 not found: ID does not exist" Nov 03 15:47:04.065888 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.065865 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:04.109225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.109194 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvmlr\" (UniqueName: \"kubernetes.io/projected/b7bd7224-6371-4352-a5ef-37bcf2155cf7-kube-api-access-cvmlr\") pod \"maas-api-revoke-sub-4bd373bc-ktlbx\" (UID: \"b7bd7224-6371-4352-a5ef-37bcf2155cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx" Nov 03 15:47:04.160395 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.160049 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" podUID="cf100201-cb1e-462e-878e-ff94690746d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:47:04.195642 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.195604 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx" Nov 03 15:47:04.202784 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.202756 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v"] Nov 03 15:47:04.206732 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.206704 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/distinct-2-w4-bc1f07-kserve-6fddf48c7f-cqh9v"] Nov 03 15:47:04.289645 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.289185 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c968070-2d65-4501-b504-e96664a97f8d" path="/var/lib/kubelet/pods/9c968070-2d65-4501-b504-e96664a97f8d/volumes" Nov 03 15:47:04.344563 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.340859 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4"] Nov 03 15:47:04.356561 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.356530 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/simulator-w4-bc1f07-kserve-c8d8dfb96-lw9s4"] Nov 03 15:47:04.614111 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:04.614089 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx"] Nov 03 15:47:04.616424 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:47:04.616398 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7bd7224_6371_4352_a5ef_37bcf2155cf7.slice/crio-bc0478a38a9339db55a1b272c1bcac54f87b8a675cab347b469bd7b17bb83c1d WatchSource:0}: Error finding container bc0478a38a9339db55a1b272c1bcac54f87b8a675cab347b469bd7b17bb83c1d: Status 404 returned error can't find the container with id bc0478a38a9339db55a1b272c1bcac54f87b8a675cab347b469bd7b17bb83c1d Nov 03 15:47:05.041170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:05.041130 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx" event={"ID":"b7bd7224-6371-4352-a5ef-37bcf2155cf7","Type":"ContainerStarted","Data":"29c94fa6ead7b19ad438192ab77e89354b285b0cf9b9d0a67c5ffcb3f5fcfb1f"} Nov 03 15:47:05.041383 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:05.041196 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx" event={"ID":"b7bd7224-6371-4352-a5ef-37bcf2155cf7","Type":"ContainerStarted","Data":"bc0478a38a9339db55a1b272c1bcac54f87b8a675cab347b469bd7b17bb83c1d"} Nov 03 15:47:05.158353 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:05.158294 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx" podStartSLOduration=2.158281142 podStartE2EDuration="2.158281142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:47:05.157373492 +0000 UTC m=+2335.000495037" watchObservedRunningTime="2026-11-03 15:47:05.158281142 +0000 UTC m=+2335.001402682" Nov 03 15:47:05.687367 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:05.687283 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:05.907186 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:05.907146 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:06.050937 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:06.050907 4557 generic.go:356] "Generic (PLEG): container finished" podID="b7bd7224-6371-4352-a5ef-37bcf2155cf7" containerID="29c94fa6ead7b19ad438192ab77e89354b285b0cf9b9d0a67c5ffcb3f5fcfb1f" exitCode=0 Nov 03 15:47:06.051134 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:06.051007 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx" event={"ID":"b7bd7224-6371-4352-a5ef-37bcf2155cf7","Type":"ContainerDied","Data":"29c94fa6ead7b19ad438192ab77e89354b285b0cf9b9d0a67c5ffcb3f5fcfb1f"} Nov 03 15:47:06.099227 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:06.099192 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:06.158707 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:06.158678 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj"] Nov 03 15:47:06.161706 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:06.161681 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e62ff3-nlhnj"] Nov 03 15:47:06.285414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:06.285384 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33449900-1ed0-43f1-88d0-422e2d5b56a2" path="/var/lib/kubelet/pods/33449900-1ed0-43f1-88d0-422e2d5b56a2/volumes" Nov 03 15:47:06.285840 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:06.285825 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc" path="/var/lib/kubelet/pods/bd48d3ec-6e5d-492f-8651-7b0bdf52c3cc/volumes" Nov 03 15:47:06.693954 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:06.693921 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:07.192320 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.192293 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx" Nov 03 15:47:07.286506 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.286474 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7bd7224-6371-4352-a5ef-37bcf2155cf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7bd7224-6371-4352-a5ef-37bcf2155cf7-maas-api-openshift-ca\") pod \"b7bd7224-6371-4352-a5ef-37bcf2155cf7\" (UID: \"b7bd7224-6371-4352-a5ef-37bcf2155cf7\") " Nov 03 15:47:07.286731 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.286530 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7bd7224-6371-4352-a5ef-37bcf2155cf7-kube-api-access-cvmlr\" (UniqueName: \"kubernetes.io/projected/b7bd7224-6371-4352-a5ef-37bcf2155cf7-kube-api-access-cvmlr\") pod \"b7bd7224-6371-4352-a5ef-37bcf2155cf7\" (UID: \"b7bd7224-6371-4352-a5ef-37bcf2155cf7\") " Nov 03 15:47:07.286731 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.286564 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7bd7224-6371-4352-a5ef-37bcf2155cf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7bd7224-6371-4352-a5ef-37bcf2155cf7-maas-api-xks-ca\") pod \"b7bd7224-6371-4352-a5ef-37bcf2155cf7\" (UID: \"b7bd7224-6371-4352-a5ef-37bcf2155cf7\") " Nov 03 15:47:07.286939 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.286915 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7bd7224-6371-4352-a5ef-37bcf2155cf7-maas-api-openshift-ca" pod "b7bd7224-6371-4352-a5ef-37bcf2155cf7" (UID: "b7bd7224-6371-4352-a5ef-37bcf2155cf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:07.288660 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.288640 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7bd7224-6371-4352-a5ef-37bcf2155cf7-maas-api-xks-ca" pod "b7bd7224-6371-4352-a5ef-37bcf2155cf7" (UID: "b7bd7224-6371-4352-a5ef-37bcf2155cf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:47:07.288717 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.288663 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7bd7224-6371-4352-a5ef-37bcf2155cf7-kube-api-access-cvmlr" pod "b7bd7224-6371-4352-a5ef-37bcf2155cf7" (UID: "b7bd7224-6371-4352-a5ef-37bcf2155cf7"). InnerVolumeSpecName "kube-api-access-cvmlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:07.387904 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.387868 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7bd7224-6371-4352-a5ef-37bcf2155cf7-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.387904 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.387901 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7bd7224-6371-4352-a5ef-37bcf2155cf7-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.388109 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.387913 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvmlr\" (UniqueName: \"kubernetes.io/projected/b7bd7224-6371-4352-a5ef-37bcf2155cf7-kube-api-access-cvmlr\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.549892 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.549867 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:47:07.589490 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.589425 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-credential-socket\") pod \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " Nov 03 15:47:07.589490 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.589469 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-workload-certs\") pod \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " Nov 03 15:47:07.589730 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.589492 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-workload-socket\") pod \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " Nov 03 15:47:07.589730 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.589524 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istiod-ca-cert\") pod \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " Nov 03 15:47:07.589730 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.589547 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37f9ec1-b784-41fd-9a79-8df424aa77e5-kube-api-access-dnmdk\" (UniqueName: \"kubernetes.io/projected/b37f9ec1-b784-41fd-9a79-8df424aa77e5-kube-api-access-dnmdk\") pod \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " Nov 03 15:47:07.589730 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.589603 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-podinfo\") pod \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " Nov 03 15:47:07.589730 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.589718 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-envoy\") pod \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " Nov 03 15:47:07.590006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.589744 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-workload-certs" pod "b37f9ec1-b784-41fd-9a79-8df424aa77e5" (UID: "b37f9ec1-b784-41fd-9a79-8df424aa77e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:07.590006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.589773 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-credential-socket" pod "b37f9ec1-b784-41fd-9a79-8df424aa77e5" (UID: "b37f9ec1-b784-41fd-9a79-8df424aa77e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:07.590006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.589791 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-data\") pod \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " Nov 03 15:47:07.590006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.589826 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-token\" (UniqueName: \"kubernetes.io/projected/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-token\") pod \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\" (UID: \"b37f9ec1-b784-41fd-9a79-8df424aa77e5\") " Nov 03 15:47:07.590006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.589935 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-workload-socket" pod "b37f9ec1-b784-41fd-9a79-8df424aa77e5" (UID: "b37f9ec1-b784-41fd-9a79-8df424aa77e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:07.590196 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.590020 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-data" pod "b37f9ec1-b784-41fd-9a79-8df424aa77e5" (UID: "b37f9ec1-b784-41fd-9a79-8df424aa77e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:07.590196 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.590036 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istiod-ca-cert" pod "b37f9ec1-b784-41fd-9a79-8df424aa77e5" (UID: "b37f9ec1-b784-41fd-9a79-8df424aa77e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:07.590319 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.590298 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.590369 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.590327 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.590369 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.590345 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.590369 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.590360 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.590488 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.590374 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.591905 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.591885 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-envoy" pod "b37f9ec1-b784-41fd-9a79-8df424aa77e5" (UID: "b37f9ec1-b784-41fd-9a79-8df424aa77e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:07.591963 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.591906 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-token" pod "b37f9ec1-b784-41fd-9a79-8df424aa77e5" (UID: "b37f9ec1-b784-41fd-9a79-8df424aa77e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:07.592203 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.592182 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-podinfo" pod "b37f9ec1-b784-41fd-9a79-8df424aa77e5" (UID: "b37f9ec1-b784-41fd-9a79-8df424aa77e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:47:07.592255 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.592183 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37f9ec1-b784-41fd-9a79-8df424aa77e5-kube-api-access-dnmdk" pod "b37f9ec1-b784-41fd-9a79-8df424aa77e5" (UID: "b37f9ec1-b784-41fd-9a79-8df424aa77e5"). InnerVolumeSpecName "kube-api-access-dnmdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:07.691553 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.691518 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnmdk\" (UniqueName: \"kubernetes.io/projected/b37f9ec1-b784-41fd-9a79-8df424aa77e5-kube-api-access-dnmdk\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.691553 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.691552 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.691867 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.691562 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:07.691867 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:07.691572 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b37f9ec1-b784-41fd-9a79-8df424aa77e5-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:08.065160 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:08.065127 4557 generic.go:356] "Generic (PLEG): container finished" podID="b37f9ec1-b784-41fd-9a79-8df424aa77e5" containerID="b7c46f81e707c8a068eeac7090cbca8557434d8e002d0dab988d451bd5ef8c7d" exitCode=0 Nov 03 15:47:08.065657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:08.065182 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" Nov 03 15:47:08.065657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:08.065219 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" event={"ID":"b37f9ec1-b784-41fd-9a79-8df424aa77e5","Type":"ContainerDied","Data":"b7c46f81e707c8a068eeac7090cbca8557434d8e002d0dab988d451bd5ef8c7d"} Nov 03 15:47:08.065657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:08.065272 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg" event={"ID":"b37f9ec1-b784-41fd-9a79-8df424aa77e5","Type":"ContainerDied","Data":"d8330a1446404bd80493b50dc016dba969f9513ba615f0460eea934021882fae"} Nov 03 15:47:08.065657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:08.065302 4557 scope.go:122] "RemoveContainer" containerID="b7c46f81e707c8a068eeac7090cbca8557434d8e002d0dab988d451bd5ef8c7d" Nov 03 15:47:08.068359 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:08.068326 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx" Nov 03 15:47:08.068359 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:08.068336 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx" event={"ID":"b7bd7224-6371-4352-a5ef-37bcf2155cf7","Type":"ContainerDied","Data":"bc0478a38a9339db55a1b272c1bcac54f87b8a675cab347b469bd7b17bb83c1d"} Nov 03 15:47:08.068536 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:08.068381 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc0478a38a9339db55a1b272c1bcac54f87b8a675cab347b469bd7b17bb83c1d" Nov 03 15:47:08.077510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:08.077452 4557 scope.go:122] "RemoveContainer" containerID="b7c46f81e707c8a068eeac7090cbca8557434d8e002d0dab988d451bd5ef8c7d" Nov 03 15:47:08.077835 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:47:08.077807 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7c46f81e707c8a068eeac7090cbca8557434d8e002d0dab988d451bd5ef8c7d\": container with ID starting with b7c46f81e707c8a068eeac7090cbca8557434d8e002d0dab988d451bd5ef8c7d not found: ID does not exist" containerID="b7c46f81e707c8a068eeac7090cbca8557434d8e002d0dab988d451bd5ef8c7d" Nov 03 15:47:08.077931 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:08.077841 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7c46f81e707c8a068eeac7090cbca8557434d8e002d0dab988d451bd5ef8c7d"} err="failed to get container status \"b7c46f81e707c8a068eeac7090cbca8557434d8e002d0dab988d451bd5ef8c7d\": rpc error: code = NotFound desc = could not find container \"b7c46f81e707c8a068eeac7090cbca8557434d8e002d0dab988d451bd5ef8c7d\": container with ID starting with b7c46f81e707c8a068eeac7090cbca8557434d8e002d0dab988d451bd5ef8c7d not found: ID does not exist" Nov 03 15:47:08.211616 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:08.211567 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:08.233014 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:08.232982 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg"] Nov 03 15:47:08.239836 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:08.239806 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7tgsg"] Nov 03 15:47:08.284833 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:08.284802 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37f9ec1-b784-41fd-9a79-8df424aa77e5" path="/var/lib/kubelet/pods/b37f9ec1-b784-41fd-9a79-8df424aa77e5/volumes" Nov 03 15:47:09.079895 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:09.079844 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:09.499042 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:09.498955 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:10.276967 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.276933 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:10.298080 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.298049 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl"] Nov 03 15:47:10.298262 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.298223 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" podUID="ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d" containerName="maas-api" containerID="cri-o://fb6db875c5ee2b1903f052eae96d7e71ac5eefb887a13c5caca145119d0875c2" gracePeriod=30 Nov 03 15:47:10.540056 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.539935 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:10.573355 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.573295 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9"] Nov 03 15:47:10.573797 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.573719 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" podUID="4386210a-32ea-4fe8-b342-63c4cc68855b" containerName="istio-proxy" containerID="cri-o://e249d06edbe300792cd0555e18253247a006c4aca0a93cafcc3c00b8420343a2" gracePeriod=30 Nov 03 15:47:10.577496 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.577470 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc"] Nov 03 15:47:10.578886 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.577780 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" podUID="f23b9ae0-fb87-47e3-b226-198a656ae1e4" containerName="istio-proxy" containerID="cri-o://e048164617d60d4efe40bc662512ce2d0ae390b76ca5a80f0a38e7f9452f514c" gracePeriod=30 Nov 03 15:47:10.579318 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.579276 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst"] Nov 03 15:47:10.579513 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.579485 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" podUID="74678262-2f6e-41a6-aa2b-00b64aa2c545" containerName="istio-proxy" containerID="cri-o://76b0eee5f7af4f60a5c71c07eea99dddc70a9c1a81fc2243cc34c0c889c9ee1f" gracePeriod=30 Nov 03 15:47:10.586306 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.586285 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:47:10.624622 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.624574 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-maas-api-tls\") pod \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\" (UID: \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\") " Nov 03 15:47:10.624797 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.624645 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-metrics-certs\") pod \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\" (UID: \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\") " Nov 03 15:47:10.624797 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.624674 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-openshift-service-ca\") pod \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\" (UID: \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\") " Nov 03 15:47:10.624797 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.624700 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-kube-api-access-djz2p\" (UniqueName: \"kubernetes.io/projected/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-kube-api-access-djz2p\") pod \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\" (UID: \"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d\") " Nov 03 15:47:10.625106 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.625069 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-openshift-service-ca" pod "ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d" (UID: "ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:10.627245 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.627214 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-kube-api-access-djz2p" pod "ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d" (UID: "ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d"). InnerVolumeSpecName "kube-api-access-djz2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:10.627339 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.627283 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-metrics-certs" pod "ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d" (UID: "ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:47:10.627339 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.627301 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-maas-api-tls" pod "ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d" (UID: "ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:47:10.726268 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.726234 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-maas-api-tls\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:10.726268 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.726267 4557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-metrics-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:10.726553 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.726281 4557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-openshift-service-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:10.726553 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.726295 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djz2p\" (UniqueName: \"kubernetes.io/projected/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d-kube-api-access-djz2p\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:10.744310 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:10.744272 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:11.088925 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:11.088896 4557 generic.go:356] "Generic (PLEG): container finished" podID="ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d" containerID="fb6db875c5ee2b1903f052eae96d7e71ac5eefb887a13c5caca145119d0875c2" exitCode=0 Nov 03 15:47:11.089075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:11.088966 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" Nov 03 15:47:11.089075 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:11.088975 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" event={"ID":"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d","Type":"ContainerDied","Data":"fb6db875c5ee2b1903f052eae96d7e71ac5eefb887a13c5caca145119d0875c2"} Nov 03 15:47:11.089178 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:11.089088 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl" event={"ID":"ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d","Type":"ContainerDied","Data":"2b744d79b2a2cd63631b202c4c7d0cfaf93e375188cc8a2535c588656ad96b71"} Nov 03 15:47:11.089178 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:11.089109 4557 scope.go:122] "RemoveContainer" containerID="fb6db875c5ee2b1903f052eae96d7e71ac5eefb887a13c5caca145119d0875c2" Nov 03 15:47:11.098968 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:11.098944 4557 scope.go:122] "RemoveContainer" containerID="fb6db875c5ee2b1903f052eae96d7e71ac5eefb887a13c5caca145119d0875c2" Nov 03 15:47:11.099295 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:47:11.099275 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6db875c5ee2b1903f052eae96d7e71ac5eefb887a13c5caca145119d0875c2\": container with ID starting with fb6db875c5ee2b1903f052eae96d7e71ac5eefb887a13c5caca145119d0875c2 not found: ID does not exist" containerID="fb6db875c5ee2b1903f052eae96d7e71ac5eefb887a13c5caca145119d0875c2" Nov 03 15:47:11.099370 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:11.099307 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6db875c5ee2b1903f052eae96d7e71ac5eefb887a13c5caca145119d0875c2"} err="failed to get container status \"fb6db875c5ee2b1903f052eae96d7e71ac5eefb887a13c5caca145119d0875c2\": rpc error: code = NotFound desc = could not find container \"fb6db875c5ee2b1903f052eae96d7e71ac5eefb887a13c5caca145119d0875c2\": container with ID starting with fb6db875c5ee2b1903f052eae96d7e71ac5eefb887a13c5caca145119d0875c2 not found: ID does not exist" Nov 03 15:47:11.266608 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:11.262615 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl"] Nov 03 15:47:11.281296 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:11.271354 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f59d26-686fb8db75-tk7nl"] Nov 03 15:47:12.012983 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:12.012924 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:12.262551 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:12.262513 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:12.284738 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:12.284670 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d" path="/var/lib/kubelet/pods/ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d/volumes" Nov 03 15:47:12.462246 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:12.462212 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:12.517824 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:12.517791 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg"] Nov 03 15:47:12.567527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:12.567495 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f59d26-fqwfg"] Nov 03 15:47:13.000282 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:13.000192 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:13.613256 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:13.613220 4557 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:47:13.613645 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:13.613282 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" podUID="f23b9ae0-fb87-47e3-b226-198a656ae1e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:47:13.644731 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:13.644693 4557 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:47:13.644893 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:13.644760 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" podUID="4386210a-32ea-4fe8-b342-63c4cc68855b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:47:14.150809 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:14.150763 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" podUID="cf100201-cb1e-462e-878e-ff94690746d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 15:47:14.285708 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:14.285672 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6428b6c6-80e9-485a-9884-f5933b41d5a8" path="/var/lib/kubelet/pods/6428b6c6-80e9-485a-9884-f5933b41d5a8/volumes" Nov 03 15:47:14.773765 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:14.773720 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:15.004225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.004175 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:15.128208 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.128170 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw"] Nov 03 15:47:15.128451 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.128383 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" podUID="f7bb3b78-db67-4de5-9b60-10d26274a199" containerName="istio-proxy" containerID="cri-o://adcdbe15e376a9ee90739e444376a9875ff4302f5555d358cc517d5aba4f5033" gracePeriod=30 Nov 03 15:47:15.212751 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.212711 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:15.832681 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.832654 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:47:15.905971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.890780 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-envoy\") pod \"4386210a-32ea-4fe8-b342-63c4cc68855b\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " Nov 03 15:47:15.905971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.890853 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-credential-socket\") pod \"4386210a-32ea-4fe8-b342-63c4cc68855b\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " Nov 03 15:47:15.905971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.890904 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-podinfo\") pod \"4386210a-32ea-4fe8-b342-63c4cc68855b\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " Nov 03 15:47:15.905971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.890979 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4386210a-32ea-4fe8-b342-63c4cc68855b-kube-api-access-hhtds\" (UniqueName: \"kubernetes.io/projected/4386210a-32ea-4fe8-b342-63c4cc68855b-kube-api-access-hhtds\") pod \"4386210a-32ea-4fe8-b342-63c4cc68855b\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " Nov 03 15:47:15.905971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.891008 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-data\") pod \"4386210a-32ea-4fe8-b342-63c4cc68855b\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " Nov 03 15:47:15.905971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.891058 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4386210a-32ea-4fe8-b342-63c4cc68855b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4386210a-32ea-4fe8-b342-63c4cc68855b-istiod-ca-cert\") pod \"4386210a-32ea-4fe8-b342-63c4cc68855b\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " Nov 03 15:47:15.905971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.891116 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-workload-socket\") pod \"4386210a-32ea-4fe8-b342-63c4cc68855b\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " Nov 03 15:47:15.905971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.891165 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-token\" (UniqueName: \"kubernetes.io/projected/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-token\") pod \"4386210a-32ea-4fe8-b342-63c4cc68855b\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " Nov 03 15:47:15.905971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.891233 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-workload-certs\") pod \"4386210a-32ea-4fe8-b342-63c4cc68855b\" (UID: \"4386210a-32ea-4fe8-b342-63c4cc68855b\") " Nov 03 15:47:15.905971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.892230 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4386210a-32ea-4fe8-b342-63c4cc68855b-istiod-ca-cert" pod "4386210a-32ea-4fe8-b342-63c4cc68855b" (UID: "4386210a-32ea-4fe8-b342-63c4cc68855b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:15.905971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.892277 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-workload-certs" pod "4386210a-32ea-4fe8-b342-63c4cc68855b" (UID: "4386210a-32ea-4fe8-b342-63c4cc68855b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:15.905971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.892484 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-workload-socket" pod "4386210a-32ea-4fe8-b342-63c4cc68855b" (UID: "4386210a-32ea-4fe8-b342-63c4cc68855b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:15.905971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.892504 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-data" pod "4386210a-32ea-4fe8-b342-63c4cc68855b" (UID: "4386210a-32ea-4fe8-b342-63c4cc68855b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:15.905971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.893073 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-credential-socket" pod "4386210a-32ea-4fe8-b342-63c4cc68855b" (UID: "4386210a-32ea-4fe8-b342-63c4cc68855b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:15.905971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.894981 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:15.905971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.898036 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-token" pod "4386210a-32ea-4fe8-b342-63c4cc68855b" (UID: "4386210a-32ea-4fe8-b342-63c4cc68855b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:15.905971 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.904484 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4386210a-32ea-4fe8-b342-63c4cc68855b-kube-api-access-hhtds" pod "4386210a-32ea-4fe8-b342-63c4cc68855b" (UID: "4386210a-32ea-4fe8-b342-63c4cc68855b"). InnerVolumeSpecName "kube-api-access-hhtds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:15.907095 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.907070 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-envoy" pod "4386210a-32ea-4fe8-b342-63c4cc68855b" (UID: "4386210a-32ea-4fe8-b342-63c4cc68855b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:15.908341 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.908229 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-podinfo" pod "4386210a-32ea-4fe8-b342-63c4cc68855b" (UID: "4386210a-32ea-4fe8-b342-63c4cc68855b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:47:15.919296 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.919276 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" Nov 03 15:47:15.922529 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.922507 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:47:15.928229 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.928213 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:47:15.991882 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.991833 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-workload-socket\") pod \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " Nov 03 15:47:15.992100 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.991903 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74678262-2f6e-41a6-aa2b-00b64aa2c545-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74678262-2f6e-41a6-aa2b-00b64aa2c545-istiod-ca-cert\") pod \"74678262-2f6e-41a6-aa2b-00b64aa2c545\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " Nov 03 15:47:15.992100 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.991932 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23b9ae0-fb87-47e3-b226-198a656ae1e4-kube-api-access-sslfc\" (UniqueName: \"kubernetes.io/projected/f23b9ae0-fb87-47e3-b226-198a656ae1e4-kube-api-access-sslfc\") pod \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " Nov 03 15:47:15.992100 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.991973 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-podinfo\") pod \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " Nov 03 15:47:15.992100 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992086 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-token\" (UniqueName: \"kubernetes.io/projected/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-token\") pod \"74678262-2f6e-41a6-aa2b-00b64aa2c545\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " Nov 03 15:47:15.992414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992105 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-workload-socket" pod "f23b9ae0-fb87-47e3-b226-198a656ae1e4" (UID: "f23b9ae0-fb87-47e3-b226-198a656ae1e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:15.992414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992166 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-credential-socket\") pod \"74678262-2f6e-41a6-aa2b-00b64aa2c545\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " Nov 03 15:47:15.992414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992234 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-credential-socket\") pod \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " Nov 03 15:47:15.992414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992268 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf100201-cb1e-462e-878e-ff94690746d7-kube-api-access-4drpg\" (UniqueName: \"kubernetes.io/projected/cf100201-cb1e-462e-878e-ff94690746d7-kube-api-access-4drpg\") pod \"cf100201-cb1e-462e-878e-ff94690746d7\" (UID: \"cf100201-cb1e-462e-878e-ff94690746d7\") " Nov 03 15:47:15.992414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992308 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-workload-certs\") pod \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " Nov 03 15:47:15.992414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992325 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74678262-2f6e-41a6-aa2b-00b64aa2c545-istiod-ca-cert" pod "74678262-2f6e-41a6-aa2b-00b64aa2c545" (UID: "74678262-2f6e-41a6-aa2b-00b64aa2c545"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:15.992414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992340 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-data\") pod \"74678262-2f6e-41a6-aa2b-00b64aa2c545\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " Nov 03 15:47:15.992414 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992384 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-workload-certs\") pod \"74678262-2f6e-41a6-aa2b-00b64aa2c545\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " Nov 03 15:47:15.992874 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992438 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-envoy\") pod \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " Nov 03 15:47:15.992874 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992477 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74678262-2f6e-41a6-aa2b-00b64aa2c545-kube-api-access-jfvrt\" (UniqueName: \"kubernetes.io/projected/74678262-2f6e-41a6-aa2b-00b64aa2c545-kube-api-access-jfvrt\") pod \"74678262-2f6e-41a6-aa2b-00b64aa2c545\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " Nov 03 15:47:15.992874 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992517 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-token\" (UniqueName: \"kubernetes.io/projected/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-token\") pod \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " Nov 03 15:47:15.992874 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992549 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-envoy\") pod \"74678262-2f6e-41a6-aa2b-00b64aa2c545\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " Nov 03 15:47:15.992874 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992597 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-podinfo\") pod \"74678262-2f6e-41a6-aa2b-00b64aa2c545\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " Nov 03 15:47:15.992874 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992606 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-credential-socket" pod "f23b9ae0-fb87-47e3-b226-198a656ae1e4" (UID: "f23b9ae0-fb87-47e3-b226-198a656ae1e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:15.992874 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992638 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-credential-socket" pod "74678262-2f6e-41a6-aa2b-00b64aa2c545" (UID: "74678262-2f6e-41a6-aa2b-00b64aa2c545"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:15.992874 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992645 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-data\") pod \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " Nov 03 15:47:15.992874 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992740 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-workload-socket\") pod \"74678262-2f6e-41a6-aa2b-00b64aa2c545\" (UID: \"74678262-2f6e-41a6-aa2b-00b64aa2c545\") " Nov 03 15:47:15.992874 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992815 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf100201-cb1e-462e-878e-ff94690746d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf100201-cb1e-462e-878e-ff94690746d7-plugins-config-volume\") pod \"cf100201-cb1e-462e-878e-ff94690746d7\" (UID: \"cf100201-cb1e-462e-878e-ff94690746d7\") " Nov 03 15:47:15.992874 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992847 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istiod-ca-cert\") pod \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\" (UID: \"f23b9ae0-fb87-47e3-b226-198a656ae1e4\") " Nov 03 15:47:15.993427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.993274 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:15.993427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.993295 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:15.993427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.993310 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74678262-2f6e-41a6-aa2b-00b64aa2c545-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:15.993427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.993326 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:15.993427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.993342 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:15.993427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.993356 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:15.993427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.993369 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:15.993427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.993385 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhtds\" (UniqueName: \"kubernetes.io/projected/4386210a-32ea-4fe8-b342-63c4cc68855b-kube-api-access-hhtds\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:15.993427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.993400 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:15.993427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.993413 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4386210a-32ea-4fe8-b342-63c4cc68855b-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:15.993427 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.993426 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:15.994049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.993440 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4386210a-32ea-4fe8-b342-63c4cc68855b-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:15.994049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.993453 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4386210a-32ea-4fe8-b342-63c4cc68855b-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:15.994049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992848 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-data" pod "f23b9ae0-fb87-47e3-b226-198a656ae1e4" (UID: "f23b9ae0-fb87-47e3-b226-198a656ae1e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:15.994049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992884 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-workload-certs" pod "74678262-2f6e-41a6-aa2b-00b64aa2c545" (UID: "74678262-2f6e-41a6-aa2b-00b64aa2c545"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:15.994049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.992974 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-data" pod "74678262-2f6e-41a6-aa2b-00b64aa2c545" (UID: "74678262-2f6e-41a6-aa2b-00b64aa2c545"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:15.994049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.993484 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf100201-cb1e-462e-878e-ff94690746d7-plugins-config-volume" pod "cf100201-cb1e-462e-878e-ff94690746d7" (UID: "cf100201-cb1e-462e-878e-ff94690746d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:15.994049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.993818 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istiod-ca-cert" pod "f23b9ae0-fb87-47e3-b226-198a656ae1e4" (UID: "f23b9ae0-fb87-47e3-b226-198a656ae1e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:15.994049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.993825 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-workload-socket" pod "74678262-2f6e-41a6-aa2b-00b64aa2c545" (UID: "74678262-2f6e-41a6-aa2b-00b64aa2c545"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:15.994049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.993954 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-workload-certs" pod "f23b9ae0-fb87-47e3-b226-198a656ae1e4" (UID: "f23b9ae0-fb87-47e3-b226-198a656ae1e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:15.995362 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.995323 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74678262-2f6e-41a6-aa2b-00b64aa2c545-kube-api-access-jfvrt" pod "74678262-2f6e-41a6-aa2b-00b64aa2c545" (UID: "74678262-2f6e-41a6-aa2b-00b64aa2c545"). InnerVolumeSpecName "kube-api-access-jfvrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:15.996094 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.996057 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-token" pod "f23b9ae0-fb87-47e3-b226-198a656ae1e4" (UID: "f23b9ae0-fb87-47e3-b226-198a656ae1e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:15.996219 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.996098 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf100201-cb1e-462e-878e-ff94690746d7-kube-api-access-4drpg" pod "cf100201-cb1e-462e-878e-ff94690746d7" (UID: "cf100201-cb1e-462e-878e-ff94690746d7"). InnerVolumeSpecName "kube-api-access-4drpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:15.996491 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.996440 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23b9ae0-fb87-47e3-b226-198a656ae1e4-kube-api-access-sslfc" pod "f23b9ae0-fb87-47e3-b226-198a656ae1e4" (UID: "f23b9ae0-fb87-47e3-b226-198a656ae1e4"). InnerVolumeSpecName "kube-api-access-sslfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:15.996491 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.996440 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-podinfo" pod "f23b9ae0-fb87-47e3-b226-198a656ae1e4" (UID: "f23b9ae0-fb87-47e3-b226-198a656ae1e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:47:15.996669 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.996559 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-podinfo" pod "74678262-2f6e-41a6-aa2b-00b64aa2c545" (UID: "74678262-2f6e-41a6-aa2b-00b64aa2c545"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:47:15.997095 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.997073 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-token" pod "74678262-2f6e-41a6-aa2b-00b64aa2c545" (UID: "74678262-2f6e-41a6-aa2b-00b64aa2c545"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:15.997272 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.997258 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-envoy" pod "74678262-2f6e-41a6-aa2b-00b64aa2c545" (UID: "74678262-2f6e-41a6-aa2b-00b64aa2c545"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:15.997833 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:15.997813 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-envoy" pod "f23b9ae0-fb87-47e3-b226-198a656ae1e4" (UID: "f23b9ae0-fb87-47e3-b226-198a656ae1e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:16.090738 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.090647 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:16.095049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.095010 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sslfc\" (UniqueName: \"kubernetes.io/projected/f23b9ae0-fb87-47e3-b226-198a656ae1e4-kube-api-access-sslfc\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.095049 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.095048 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.095287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.095063 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.095287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.095077 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4drpg\" (UniqueName: \"kubernetes.io/projected/cf100201-cb1e-462e-878e-ff94690746d7-kube-api-access-4drpg\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.095287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.095092 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.095287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.095106 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.095287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.095119 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.095287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.095132 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.095287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.095146 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfvrt\" (UniqueName: \"kubernetes.io/projected/74678262-2f6e-41a6-aa2b-00b64aa2c545-kube-api-access-jfvrt\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.095287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.095159 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.095287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.095171 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.095287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.095187 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74678262-2f6e-41a6-aa2b-00b64aa2c545-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.095287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.095201 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.095287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.095215 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74678262-2f6e-41a6-aa2b-00b64aa2c545-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.095287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.095229 4557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf100201-cb1e-462e-878e-ff94690746d7-plugins-config-volume\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.095287 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.095243 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23b9ae0-fb87-47e3-b226-198a656ae1e4-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:16.123078 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.123046 4557 generic.go:356] "Generic (PLEG): container finished" podID="74678262-2f6e-41a6-aa2b-00b64aa2c545" containerID="76b0eee5f7af4f60a5c71c07eea99dddc70a9c1a81fc2243cc34c0c889c9ee1f" exitCode=0 Nov 03 15:47:16.123267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.123101 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" Nov 03 15:47:16.123267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.123146 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" event={"ID":"74678262-2f6e-41a6-aa2b-00b64aa2c545","Type":"ContainerDied","Data":"76b0eee5f7af4f60a5c71c07eea99dddc70a9c1a81fc2243cc34c0c889c9ee1f"} Nov 03 15:47:16.123267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.123223 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst" event={"ID":"74678262-2f6e-41a6-aa2b-00b64aa2c545","Type":"ContainerDied","Data":"c3f56b000a0fdf20cb9c89e740412de5865ba5e66da1243eefea75c94eb6b251"} Nov 03 15:47:16.123267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.123259 4557 scope.go:122] "RemoveContainer" containerID="76b0eee5f7af4f60a5c71c07eea99dddc70a9c1a81fc2243cc34c0c889c9ee1f" Nov 03 15:47:16.124736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.124710 4557 generic.go:356] "Generic (PLEG): container finished" podID="4386210a-32ea-4fe8-b342-63c4cc68855b" containerID="e249d06edbe300792cd0555e18253247a006c4aca0a93cafcc3c00b8420343a2" exitCode=0 Nov 03 15:47:16.124868 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.124771 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" Nov 03 15:47:16.124868 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.124793 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" event={"ID":"4386210a-32ea-4fe8-b342-63c4cc68855b","Type":"ContainerDied","Data":"e249d06edbe300792cd0555e18253247a006c4aca0a93cafcc3c00b8420343a2"} Nov 03 15:47:16.124868 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.124859 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9" event={"ID":"4386210a-32ea-4fe8-b342-63c4cc68855b","Type":"ContainerDied","Data":"96280fb1467ad6e19729ccf446e5225a78da5e5c566970701d7da5e189586f2b"} Nov 03 15:47:16.126635 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.126573 4557 generic.go:356] "Generic (PLEG): container finished" podID="f23b9ae0-fb87-47e3-b226-198a656ae1e4" containerID="e048164617d60d4efe40bc662512ce2d0ae390b76ca5a80f0a38e7f9452f514c" exitCode=0 Nov 03 15:47:16.126741 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.126712 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" event={"ID":"f23b9ae0-fb87-47e3-b226-198a656ae1e4","Type":"ContainerDied","Data":"e048164617d60d4efe40bc662512ce2d0ae390b76ca5a80f0a38e7f9452f514c"} Nov 03 15:47:16.126819 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.126752 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" event={"ID":"f23b9ae0-fb87-47e3-b226-198a656ae1e4","Type":"ContainerDied","Data":"2838c3c83f090417009b03e6eefae3ce556eb91adce8970cca6d7ef7f6696bb7"} Nov 03 15:47:16.126819 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.126816 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc" Nov 03 15:47:16.132284 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.132259 4557 generic.go:356] "Generic (PLEG): container finished" podID="cf100201-cb1e-462e-878e-ff94690746d7" containerID="5f1886533aae05aaf35fe19bbc428c70c2101e18c9cf3a143f5e96c86f6cdf46" exitCode=0 Nov 03 15:47:16.132428 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.132302 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" event={"ID":"cf100201-cb1e-462e-878e-ff94690746d7","Type":"ContainerDied","Data":"5f1886533aae05aaf35fe19bbc428c70c2101e18c9cf3a143f5e96c86f6cdf46"} Nov 03 15:47:16.132428 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.132341 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" event={"ID":"cf100201-cb1e-462e-878e-ff94690746d7","Type":"ContainerDied","Data":"6cf5baef47898a5d3d838f1e890ad67cfe18da387502760dda3f8536516c532a"} Nov 03 15:47:16.132428 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.132408 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7" Nov 03 15:47:16.134197 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.134181 4557 scope.go:122] "RemoveContainer" containerID="8f98e891a9fd39a67d070baca9938c500b6259f4491099f479f7f973c4ca8b32" Nov 03 15:47:16.146661 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.146638 4557 scope.go:122] "RemoveContainer" containerID="76b0eee5f7af4f60a5c71c07eea99dddc70a9c1a81fc2243cc34c0c889c9ee1f" Nov 03 15:47:16.146968 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:47:16.146951 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b0eee5f7af4f60a5c71c07eea99dddc70a9c1a81fc2243cc34c0c889c9ee1f\": container with ID starting with 76b0eee5f7af4f60a5c71c07eea99dddc70a9c1a81fc2243cc34c0c889c9ee1f not found: ID does not exist" containerID="76b0eee5f7af4f60a5c71c07eea99dddc70a9c1a81fc2243cc34c0c889c9ee1f" Nov 03 15:47:16.147033 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.146977 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b0eee5f7af4f60a5c71c07eea99dddc70a9c1a81fc2243cc34c0c889c9ee1f"} err="failed to get container status \"76b0eee5f7af4f60a5c71c07eea99dddc70a9c1a81fc2243cc34c0c889c9ee1f\": rpc error: code = NotFound desc = could not find container \"76b0eee5f7af4f60a5c71c07eea99dddc70a9c1a81fc2243cc34c0c889c9ee1f\": container with ID starting with 76b0eee5f7af4f60a5c71c07eea99dddc70a9c1a81fc2243cc34c0c889c9ee1f not found: ID does not exist" Nov 03 15:47:16.147033 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.146994 4557 scope.go:122] "RemoveContainer" containerID="8f98e891a9fd39a67d070baca9938c500b6259f4491099f479f7f973c4ca8b32" Nov 03 15:47:16.147306 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:47:16.147285 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f98e891a9fd39a67d070baca9938c500b6259f4491099f479f7f973c4ca8b32\": container with ID starting with 8f98e891a9fd39a67d070baca9938c500b6259f4491099f479f7f973c4ca8b32 not found: ID does not exist" containerID="8f98e891a9fd39a67d070baca9938c500b6259f4491099f479f7f973c4ca8b32" Nov 03 15:47:16.147394 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.147317 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f98e891a9fd39a67d070baca9938c500b6259f4491099f479f7f973c4ca8b32"} err="failed to get container status \"8f98e891a9fd39a67d070baca9938c500b6259f4491099f479f7f973c4ca8b32\": rpc error: code = NotFound desc = could not find container \"8f98e891a9fd39a67d070baca9938c500b6259f4491099f479f7f973c4ca8b32\": container with ID starting with 8f98e891a9fd39a67d070baca9938c500b6259f4491099f479f7f973c4ca8b32 not found: ID does not exist" Nov 03 15:47:16.147394 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.147339 4557 scope.go:122] "RemoveContainer" containerID="e249d06edbe300792cd0555e18253247a006c4aca0a93cafcc3c00b8420343a2" Nov 03 15:47:16.156471 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.156448 4557 scope.go:122] "RemoveContainer" containerID="e249d06edbe300792cd0555e18253247a006c4aca0a93cafcc3c00b8420343a2" Nov 03 15:47:16.156817 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:47:16.156789 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e249d06edbe300792cd0555e18253247a006c4aca0a93cafcc3c00b8420343a2\": container with ID starting with e249d06edbe300792cd0555e18253247a006c4aca0a93cafcc3c00b8420343a2 not found: ID does not exist" containerID="e249d06edbe300792cd0555e18253247a006c4aca0a93cafcc3c00b8420343a2" Nov 03 15:47:16.156913 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.156816 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e249d06edbe300792cd0555e18253247a006c4aca0a93cafcc3c00b8420343a2"} err="failed to get container status \"e249d06edbe300792cd0555e18253247a006c4aca0a93cafcc3c00b8420343a2\": rpc error: code = NotFound desc = could not find container \"e249d06edbe300792cd0555e18253247a006c4aca0a93cafcc3c00b8420343a2\": container with ID starting with e249d06edbe300792cd0555e18253247a006c4aca0a93cafcc3c00b8420343a2 not found: ID does not exist" Nov 03 15:47:16.156913 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.156834 4557 scope.go:122] "RemoveContainer" containerID="e048164617d60d4efe40bc662512ce2d0ae390b76ca5a80f0a38e7f9452f514c" Nov 03 15:47:16.168807 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.168782 4557 scope.go:122] "RemoveContainer" containerID="e048164617d60d4efe40bc662512ce2d0ae390b76ca5a80f0a38e7f9452f514c" Nov 03 15:47:16.169187 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:47:16.169162 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e048164617d60d4efe40bc662512ce2d0ae390b76ca5a80f0a38e7f9452f514c\": container with ID starting with e048164617d60d4efe40bc662512ce2d0ae390b76ca5a80f0a38e7f9452f514c not found: ID does not exist" containerID="e048164617d60d4efe40bc662512ce2d0ae390b76ca5a80f0a38e7f9452f514c" Nov 03 15:47:16.169259 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.169198 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e048164617d60d4efe40bc662512ce2d0ae390b76ca5a80f0a38e7f9452f514c"} err="failed to get container status \"e048164617d60d4efe40bc662512ce2d0ae390b76ca5a80f0a38e7f9452f514c\": rpc error: code = NotFound desc = could not find container \"e048164617d60d4efe40bc662512ce2d0ae390b76ca5a80f0a38e7f9452f514c\": container with ID starting with e048164617d60d4efe40bc662512ce2d0ae390b76ca5a80f0a38e7f9452f514c not found: ID does not exist" Nov 03 15:47:16.169259 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.169223 4557 scope.go:122] "RemoveContainer" containerID="5f1886533aae05aaf35fe19bbc428c70c2101e18c9cf3a143f5e96c86f6cdf46" Nov 03 15:47:16.177231 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.177213 4557 scope.go:122] "RemoveContainer" containerID="5f1886533aae05aaf35fe19bbc428c70c2101e18c9cf3a143f5e96c86f6cdf46" Nov 03 15:47:16.177511 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:47:16.177487 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1886533aae05aaf35fe19bbc428c70c2101e18c9cf3a143f5e96c86f6cdf46\": container with ID starting with 5f1886533aae05aaf35fe19bbc428c70c2101e18c9cf3a143f5e96c86f6cdf46 not found: ID does not exist" containerID="5f1886533aae05aaf35fe19bbc428c70c2101e18c9cf3a143f5e96c86f6cdf46" Nov 03 15:47:16.177559 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.177525 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1886533aae05aaf35fe19bbc428c70c2101e18c9cf3a143f5e96c86f6cdf46"} err="failed to get container status \"5f1886533aae05aaf35fe19bbc428c70c2101e18c9cf3a143f5e96c86f6cdf46\": rpc error: code = NotFound desc = could not find container \"5f1886533aae05aaf35fe19bbc428c70c2101e18c9cf3a143f5e96c86f6cdf46\": container with ID starting with 5f1886533aae05aaf35fe19bbc428c70c2101e18c9cf3a143f5e96c86f6cdf46 not found: ID does not exist" Nov 03 15:47:16.446796 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.446720 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst"] Nov 03 15:47:16.455117 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.455091 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-9wpst"] Nov 03 15:47:16.530028 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:47:16.529989 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:47:16.566170 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.566138 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc"] Nov 03 15:47:16.572329 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.572192 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-cblsc"] Nov 03 15:47:16.683944 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.683913 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9"] Nov 03 15:47:16.689602 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.689557 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-qz8h9"] Nov 03 15:47:16.800619 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.800592 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7"] Nov 03 15:47:16.805718 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.805693 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e62ff3-7f6c89fdcf-zjmt7"] Nov 03 15:47:16.963854 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:16.963828 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:18.298022 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:18.297984 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4386210a-32ea-4fe8-b342-63c4cc68855b" path="/var/lib/kubelet/pods/4386210a-32ea-4fe8-b342-63c4cc68855b/volumes" Nov 03 15:47:18.298639 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:18.298620 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74678262-2f6e-41a6-aa2b-00b64aa2c545" path="/var/lib/kubelet/pods/74678262-2f6e-41a6-aa2b-00b64aa2c545/volumes" Nov 03 15:47:18.299132 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:18.299106 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf100201-cb1e-462e-878e-ff94690746d7" path="/var/lib/kubelet/pods/cf100201-cb1e-462e-878e-ff94690746d7/volumes" Nov 03 15:47:18.299860 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:18.299840 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f23b9ae0-fb87-47e3-b226-198a656ae1e4" path="/var/lib/kubelet/pods/f23b9ae0-fb87-47e3-b226-198a656ae1e4/volumes" Nov 03 15:47:18.519280 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:18.519241 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:18.799265 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:18.799003 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:18.994042 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:18.994006 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf"] Nov 03 15:47:19.043876 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:19.043847 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bc1f07-wqjrf"] Nov 03 15:47:19.536649 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:19.536617 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:20.285212 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.285177 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ac71731-3b40-418b-8ddf-816a8aafd5e7" path="/var/lib/kubelet/pods/2ac71731-3b40-418b-8ddf-816a8aafd5e7/volumes" Nov 03 15:47:20.383554 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.383526 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:47:20.547678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.539877 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7bb3b78-db67-4de5-9b60-10d26274a199-kube-api-access-97lzj\" (UniqueName: \"kubernetes.io/projected/f7bb3b78-db67-4de5-9b60-10d26274a199-kube-api-access-97lzj\") pod \"f7bb3b78-db67-4de5-9b60-10d26274a199\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " Nov 03 15:47:20.547678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.539947 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-token\" (UniqueName: \"kubernetes.io/projected/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-token\") pod \"f7bb3b78-db67-4de5-9b60-10d26274a199\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " Nov 03 15:47:20.547678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.539997 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-podinfo\") pod \"f7bb3b78-db67-4de5-9b60-10d26274a199\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " Nov 03 15:47:20.547678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.540056 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-workload-certs\") pod \"f7bb3b78-db67-4de5-9b60-10d26274a199\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " Nov 03 15:47:20.547678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.540084 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-credential-socket\") pod \"f7bb3b78-db67-4de5-9b60-10d26274a199\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " Nov 03 15:47:20.547678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.540155 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-envoy\") pod \"f7bb3b78-db67-4de5-9b60-10d26274a199\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " Nov 03 15:47:20.547678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.540185 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-workload-socket\") pod \"f7bb3b78-db67-4de5-9b60-10d26274a199\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " Nov 03 15:47:20.547678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.540282 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7bb3b78-db67-4de5-9b60-10d26274a199-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7bb3b78-db67-4de5-9b60-10d26274a199-istiod-ca-cert\") pod \"f7bb3b78-db67-4de5-9b60-10d26274a199\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " Nov 03 15:47:20.547678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.540317 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-data\") pod \"f7bb3b78-db67-4de5-9b60-10d26274a199\" (UID: \"f7bb3b78-db67-4de5-9b60-10d26274a199\") " Nov 03 15:47:20.547678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.540960 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-data" pod "f7bb3b78-db67-4de5-9b60-10d26274a199" (UID: "f7bb3b78-db67-4de5-9b60-10d26274a199"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:20.547678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.542297 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-workload-socket" pod "f7bb3b78-db67-4de5-9b60-10d26274a199" (UID: "f7bb3b78-db67-4de5-9b60-10d26274a199"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:20.547678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.542403 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7bb3b78-db67-4de5-9b60-10d26274a199-istiod-ca-cert" pod "f7bb3b78-db67-4de5-9b60-10d26274a199" (UID: "f7bb3b78-db67-4de5-9b60-10d26274a199"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:20.547678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.542528 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-credential-socket" pod "f7bb3b78-db67-4de5-9b60-10d26274a199" (UID: "f7bb3b78-db67-4de5-9b60-10d26274a199"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:20.547678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.542767 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-workload-certs" pod "f7bb3b78-db67-4de5-9b60-10d26274a199" (UID: "f7bb3b78-db67-4de5-9b60-10d26274a199"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:20.547678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.544987 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-token" pod "f7bb3b78-db67-4de5-9b60-10d26274a199" (UID: "f7bb3b78-db67-4de5-9b60-10d26274a199"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:20.547678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.547011 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-envoy" pod "f7bb3b78-db67-4de5-9b60-10d26274a199" (UID: "f7bb3b78-db67-4de5-9b60-10d26274a199"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:20.547678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.547116 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bb3b78-db67-4de5-9b60-10d26274a199-kube-api-access-97lzj" pod "f7bb3b78-db67-4de5-9b60-10d26274a199" (UID: "f7bb3b78-db67-4de5-9b60-10d26274a199"). InnerVolumeSpecName "kube-api-access-97lzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:20.558859 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.558799 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-podinfo" pod "f7bb3b78-db67-4de5-9b60-10d26274a199" (UID: "f7bb3b78-db67-4de5-9b60-10d26274a199"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:47:20.641410 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.641375 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.641410 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.641408 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.641653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.641419 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7bb3b78-db67-4de5-9b60-10d26274a199-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.641653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.641428 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.641653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.641437 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97lzj\" (UniqueName: \"kubernetes.io/projected/f7bb3b78-db67-4de5-9b60-10d26274a199-kube-api-access-97lzj\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.641653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.641447 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.641653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.641456 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7bb3b78-db67-4de5-9b60-10d26274a199-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.641653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.641464 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:20.641653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:20.641472 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb3b78-db67-4de5-9b60-10d26274a199-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:21.121451 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.121409 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:21.159357 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.159312 4557 generic.go:356] "Generic (PLEG): container finished" podID="f7bb3b78-db67-4de5-9b60-10d26274a199" containerID="adcdbe15e376a9ee90739e444376a9875ff4302f5555d358cc517d5aba4f5033" exitCode=0 Nov 03 15:47:21.159518 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.159396 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" event={"ID":"f7bb3b78-db67-4de5-9b60-10d26274a199","Type":"ContainerDied","Data":"adcdbe15e376a9ee90739e444376a9875ff4302f5555d358cc517d5aba4f5033"} Nov 03 15:47:21.159518 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.159402 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" Nov 03 15:47:21.159518 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.159448 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" event={"ID":"f7bb3b78-db67-4de5-9b60-10d26274a199","Type":"ContainerDied","Data":"7294eda2670fe3554672ccf453ecef55239b0c5b06e63fd9e5bb1d5f21677b53"} Nov 03 15:47:21.159518 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.159472 4557 scope.go:122] "RemoveContainer" containerID="adcdbe15e376a9ee90739e444376a9875ff4302f5555d358cc517d5aba4f5033" Nov 03 15:47:21.169890 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.169867 4557 scope.go:122] "RemoveContainer" containerID="da8a589ec74bd3921a6aad779ff7607ca10de1c21d0ce5fba5c484f9e57c13bc" Nov 03 15:47:21.178335 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.178309 4557 scope.go:122] "RemoveContainer" containerID="adcdbe15e376a9ee90739e444376a9875ff4302f5555d358cc517d5aba4f5033" Nov 03 15:47:21.178663 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:47:21.178640 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adcdbe15e376a9ee90739e444376a9875ff4302f5555d358cc517d5aba4f5033\": container with ID starting with adcdbe15e376a9ee90739e444376a9875ff4302f5555d358cc517d5aba4f5033 not found: ID does not exist" containerID="adcdbe15e376a9ee90739e444376a9875ff4302f5555d358cc517d5aba4f5033" Nov 03 15:47:21.178742 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.178670 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adcdbe15e376a9ee90739e444376a9875ff4302f5555d358cc517d5aba4f5033"} err="failed to get container status \"adcdbe15e376a9ee90739e444376a9875ff4302f5555d358cc517d5aba4f5033\": rpc error: code = NotFound desc = could not find container \"adcdbe15e376a9ee90739e444376a9875ff4302f5555d358cc517d5aba4f5033\": container with ID starting with adcdbe15e376a9ee90739e444376a9875ff4302f5555d358cc517d5aba4f5033 not found: ID does not exist" Nov 03 15:47:21.178742 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.178689 4557 scope.go:122] "RemoveContainer" containerID="da8a589ec74bd3921a6aad779ff7607ca10de1c21d0ce5fba5c484f9e57c13bc" Nov 03 15:47:21.178942 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:47:21.178925 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da8a589ec74bd3921a6aad779ff7607ca10de1c21d0ce5fba5c484f9e57c13bc\": container with ID starting with da8a589ec74bd3921a6aad779ff7607ca10de1c21d0ce5fba5c484f9e57c13bc not found: ID does not exist" containerID="da8a589ec74bd3921a6aad779ff7607ca10de1c21d0ce5fba5c484f9e57c13bc" Nov 03 15:47:21.178982 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.178949 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8a589ec74bd3921a6aad779ff7607ca10de1c21d0ce5fba5c484f9e57c13bc"} err="failed to get container status \"da8a589ec74bd3921a6aad779ff7607ca10de1c21d0ce5fba5c484f9e57c13bc\": rpc error: code = NotFound desc = could not find container \"da8a589ec74bd3921a6aad779ff7607ca10de1c21d0ce5fba5c484f9e57c13bc\": container with ID starting with da8a589ec74bd3921a6aad779ff7607ca10de1c21d0ce5fba5c484f9e57c13bc not found: ID does not exist" Nov 03 15:47:21.324634 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.324576 4557 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:47:21.324799 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.324660 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw" podUID="f7bb3b78-db67-4de5-9b60-10d26274a199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:47:21.333474 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.333445 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:21.377001 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.376911 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw"] Nov 03 15:47:21.427754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.427711 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-2vzvw"] Nov 03 15:47:21.464473 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.464443 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8"] Nov 03 15:47:21.464702 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.464677 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" podUID="7620cddf-1590-4162-a61a-71fad3ef52ae" containerName="istio-proxy" containerID="cri-o://eca0b29d091730481266ab847eab93d1459da0c8e339131bdf23761ee5afa4be" gracePeriod=30 Nov 03 15:47:21.568739 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:21.568701 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:22.284133 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:22.284102 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7bb3b78-db67-4de5-9b60-10d26274a199" path="/var/lib/kubelet/pods/f7bb3b78-db67-4de5-9b60-10d26274a199/volumes" Nov 03 15:47:23.220162 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:23.220127 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:23.450718 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:23.450678 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:24.073094 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:24.073049 4557 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:47:24.073260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:24.073112 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" podUID="7620cddf-1590-4162-a61a-71fad3ef52ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:47:24.108027 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:24.107693 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:26.731714 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.731654 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:47:26.849269 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.849224 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:26.913029 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.907935 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-workload-socket\") pod \"7620cddf-1590-4162-a61a-71fad3ef52ae\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " Nov 03 15:47:26.913029 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.908007 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-data\") pod \"7620cddf-1590-4162-a61a-71fad3ef52ae\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " Nov 03 15:47:26.913029 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.908048 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-envoy\") pod \"7620cddf-1590-4162-a61a-71fad3ef52ae\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " Nov 03 15:47:26.913029 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.908093 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-podinfo\") pod \"7620cddf-1590-4162-a61a-71fad3ef52ae\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " Nov 03 15:47:26.913029 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.908137 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-token\" (UniqueName: \"kubernetes.io/projected/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-token\") pod \"7620cddf-1590-4162-a61a-71fad3ef52ae\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " Nov 03 15:47:26.913029 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.908184 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-credential-socket\") pod \"7620cddf-1590-4162-a61a-71fad3ef52ae\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " Nov 03 15:47:26.913029 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.908231 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7620cddf-1590-4162-a61a-71fad3ef52ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7620cddf-1590-4162-a61a-71fad3ef52ae-istiod-ca-cert\") pod \"7620cddf-1590-4162-a61a-71fad3ef52ae\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " Nov 03 15:47:26.913029 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.908311 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7620cddf-1590-4162-a61a-71fad3ef52ae-kube-api-access-pl8dn\" (UniqueName: \"kubernetes.io/projected/7620cddf-1590-4162-a61a-71fad3ef52ae-kube-api-access-pl8dn\") pod \"7620cddf-1590-4162-a61a-71fad3ef52ae\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " Nov 03 15:47:26.913029 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.908358 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-workload-certs\") pod \"7620cddf-1590-4162-a61a-71fad3ef52ae\" (UID: \"7620cddf-1590-4162-a61a-71fad3ef52ae\") " Nov 03 15:47:26.914942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.914822 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-workload-certs" pod "7620cddf-1590-4162-a61a-71fad3ef52ae" (UID: "7620cddf-1590-4162-a61a-71fad3ef52ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:26.914942 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.914904 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-workload-socket" pod "7620cddf-1590-4162-a61a-71fad3ef52ae" (UID: "7620cddf-1590-4162-a61a-71fad3ef52ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:26.915113 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.915079 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-data" pod "7620cddf-1590-4162-a61a-71fad3ef52ae" (UID: "7620cddf-1590-4162-a61a-71fad3ef52ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:26.915298 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.915273 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:26.915391 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.915302 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:26.915391 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.915316 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:26.915540 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.915522 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-credential-socket" pod "7620cddf-1590-4162-a61a-71fad3ef52ae" (UID: "7620cddf-1590-4162-a61a-71fad3ef52ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:26.916304 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.916283 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7620cddf-1590-4162-a61a-71fad3ef52ae-istiod-ca-cert" pod "7620cddf-1590-4162-a61a-71fad3ef52ae" (UID: "7620cddf-1590-4162-a61a-71fad3ef52ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:47:26.928115 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.928085 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-podinfo" pod "7620cddf-1590-4162-a61a-71fad3ef52ae" (UID: "7620cddf-1590-4162-a61a-71fad3ef52ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:47:26.929601 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.929562 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7620cddf-1590-4162-a61a-71fad3ef52ae-kube-api-access-pl8dn" pod "7620cddf-1590-4162-a61a-71fad3ef52ae" (UID: "7620cddf-1590-4162-a61a-71fad3ef52ae"). InnerVolumeSpecName "kube-api-access-pl8dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:26.931476 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.931279 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-envoy" pod "7620cddf-1590-4162-a61a-71fad3ef52ae" (UID: "7620cddf-1590-4162-a61a-71fad3ef52ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:47:26.931972 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:26.931951 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-token" pod "7620cddf-1590-4162-a61a-71fad3ef52ae" (UID: "7620cddf-1590-4162-a61a-71fad3ef52ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:47:27.016527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.016455 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:27.016527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.016481 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:27.016527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.016494 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7620cddf-1590-4162-a61a-71fad3ef52ae-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:27.016527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.016505 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620cddf-1590-4162-a61a-71fad3ef52ae-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:27.016527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.016513 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7620cddf-1590-4162-a61a-71fad3ef52ae-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:27.016527 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.016523 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl8dn\" (UniqueName: \"kubernetes.io/projected/7620cddf-1590-4162-a61a-71fad3ef52ae-kube-api-access-pl8dn\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:47:27.047309 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.047283 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:27.189977 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.189944 4557 generic.go:356] "Generic (PLEG): container finished" podID="7620cddf-1590-4162-a61a-71fad3ef52ae" containerID="eca0b29d091730481266ab847eab93d1459da0c8e339131bdf23761ee5afa4be" exitCode=0 Nov 03 15:47:27.190150 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.189991 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" Nov 03 15:47:27.190150 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.189997 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" event={"ID":"7620cddf-1590-4162-a61a-71fad3ef52ae","Type":"ContainerDied","Data":"eca0b29d091730481266ab847eab93d1459da0c8e339131bdf23761ee5afa4be"} Nov 03 15:47:27.190150 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.190124 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8" event={"ID":"7620cddf-1590-4162-a61a-71fad3ef52ae","Type":"ContainerDied","Data":"36c3de1383e316c7398c1e667415ab2cbef46ff21ef40ef4920216d31c2e7286"} Nov 03 15:47:27.190249 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.190153 4557 scope.go:122] "RemoveContainer" containerID="eca0b29d091730481266ab847eab93d1459da0c8e339131bdf23761ee5afa4be" Nov 03 15:47:27.199473 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.199455 4557 scope.go:122] "RemoveContainer" containerID="eca0b29d091730481266ab847eab93d1459da0c8e339131bdf23761ee5afa4be" Nov 03 15:47:27.199773 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:47:27.199747 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eca0b29d091730481266ab847eab93d1459da0c8e339131bdf23761ee5afa4be\": container with ID starting with eca0b29d091730481266ab847eab93d1459da0c8e339131bdf23761ee5afa4be not found: ID does not exist" containerID="eca0b29d091730481266ab847eab93d1459da0c8e339131bdf23761ee5afa4be" Nov 03 15:47:27.199833 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.199781 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca0b29d091730481266ab847eab93d1459da0c8e339131bdf23761ee5afa4be"} err="failed to get container status \"eca0b29d091730481266ab847eab93d1459da0c8e339131bdf23761ee5afa4be\": rpc error: code = NotFound desc = could not find container \"eca0b29d091730481266ab847eab93d1459da0c8e339131bdf23761ee5afa4be\": container with ID starting with eca0b29d091730481266ab847eab93d1459da0c8e339131bdf23761ee5afa4be not found: ID does not exist" Nov 03 15:47:27.241171 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.241129 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:27.356757 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.356722 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8"] Nov 03 15:47:27.406429 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.406397 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-sw7f8"] Nov 03 15:47:27.700572 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.700486 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:27.899773 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:27.899743 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:28.111683 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:28.111645 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:28.284361 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:28.284327 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7620cddf-1590-4162-a61a-71fad3ef52ae" path="/var/lib/kubelet/pods/7620cddf-1590-4162-a61a-71fad3ef52ae/volumes" Nov 03 15:47:29.146592 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:29.146548 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:34.199609 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:34.199074 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:36.995267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:36.995230 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:39.405526 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:39.405491 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:39.616774 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:39.616737 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:44.345004 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:44.344971 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:45.132137 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:45.132105 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:46.051860 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:46.051828 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:46.528615 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:47:46.528512 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:47:49.304985 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:49.304939 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:54.330096 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:54.330010 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:47:59.319033 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:47:59.318995 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:48:04.213163 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:04.213046 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:48:05.946443 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:05.946407 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:48:09.119278 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:09.119235 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:48:09.434020 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:09.433934 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:48:10.593467 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:10.593432 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:48:10.600569 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:10.600552 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:48:14.017211 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:14.017166 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:48:16.528804 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:48:16.528769 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:48:19.191713 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:19.191680 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:48:20.274320 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:20.274287 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:48:24.408412 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:24.408374 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:48:25.194604 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:25.188826 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg"] Nov 03 15:48:25.439183 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:25.439148 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:48:25.469792 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:25.469690 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5952j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:48:25.469792 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:25.469764 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" podUID="6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:48:29.530154 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:29.530122 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:48:32.705207 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:32.705156 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:48:32.919754 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:32.919704 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:48:34.516872 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:34.516827 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:48:38.847839 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.847797 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc"] Nov 03 15:48:38.848260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848181 4557 cpu_manager.go:414] "containerMap: removing container" podUID="b37f9ec1-b784-41fd-9a79-8df424aa77e5" containerName="istio-proxy" Nov 03 15:48:38.848260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848193 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37f9ec1-b784-41fd-9a79-8df424aa77e5" containerName="istio-proxy" Nov 03 15:48:38.848260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848207 4557 cpu_manager.go:414] "containerMap: removing container" podUID="74678262-2f6e-41a6-aa2b-00b64aa2c545" containerName="istio-proxy" Nov 03 15:48:38.848260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848212 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74678262-2f6e-41a6-aa2b-00b64aa2c545" containerName="istio-proxy" Nov 03 15:48:38.848260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848222 4557 cpu_manager.go:414] "containerMap: removing container" podUID="cf100201-cb1e-462e-878e-ff94690746d7" containerName="payload-processing" Nov 03 15:48:38.848260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848228 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf100201-cb1e-462e-878e-ff94690746d7" containerName="payload-processing" Nov 03 15:48:38.848260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848238 4557 cpu_manager.go:414] "containerMap: removing container" podUID="f7bb3b78-db67-4de5-9b60-10d26274a199" containerName="istio-proxy" Nov 03 15:48:38.848260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848245 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7bb3b78-db67-4de5-9b60-10d26274a199" containerName="istio-proxy" Nov 03 15:48:38.848260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848254 4557 cpu_manager.go:414] "containerMap: removing container" podUID="b7bd7224-6371-4352-a5ef-37bcf2155cf7" containerName="revoke-keys" Nov 03 15:48:38.848260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848259 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7bd7224-6371-4352-a5ef-37bcf2155cf7" containerName="revoke-keys" Nov 03 15:48:38.848260 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848265 4557 cpu_manager.go:414] "containerMap: removing container" podUID="7620cddf-1590-4162-a61a-71fad3ef52ae" containerName="istio-proxy" Nov 03 15:48:38.848665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848270 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7620cddf-1590-4162-a61a-71fad3ef52ae" containerName="istio-proxy" Nov 03 15:48:38.848665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848278 4557 cpu_manager.go:414] "containerMap: removing container" podUID="4386210a-32ea-4fe8-b342-63c4cc68855b" containerName="istio-proxy" Nov 03 15:48:38.848665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848282 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4386210a-32ea-4fe8-b342-63c4cc68855b" containerName="istio-proxy" Nov 03 15:48:38.848665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848292 4557 cpu_manager.go:414] "containerMap: removing container" podUID="f23b9ae0-fb87-47e3-b226-198a656ae1e4" containerName="istio-proxy" Nov 03 15:48:38.848665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848296 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23b9ae0-fb87-47e3-b226-198a656ae1e4" containerName="istio-proxy" Nov 03 15:48:38.848665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848304 4557 cpu_manager.go:414] "containerMap: removing container" podUID="ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d" containerName="maas-api" Nov 03 15:48:38.848665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848308 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d" containerName="maas-api" Nov 03 15:48:38.848665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848381 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="74678262-2f6e-41a6-aa2b-00b64aa2c545" containerName="istio-proxy" Nov 03 15:48:38.848665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848389 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf100201-cb1e-462e-878e-ff94690746d7" containerName="payload-processing" Nov 03 15:48:38.848665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848398 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7bb3b78-db67-4de5-9b60-10d26274a199" containerName="istio-proxy" Nov 03 15:48:38.848665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848407 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="7620cddf-1590-4162-a61a-71fad3ef52ae" containerName="istio-proxy" Nov 03 15:48:38.848665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848414 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7bd7224-6371-4352-a5ef-37bcf2155cf7" containerName="revoke-keys" Nov 03 15:48:38.848665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848421 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="4386210a-32ea-4fe8-b342-63c4cc68855b" containerName="istio-proxy" Nov 03 15:48:38.848665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848427 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23b9ae0-fb87-47e3-b226-198a656ae1e4" containerName="istio-proxy" Nov 03 15:48:38.848665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848434 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7bb3b78-db67-4de5-9b60-10d26274a199" containerName="istio-proxy" Nov 03 15:48:38.848665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848439 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6782e4-9efb-4d59-b7e3-c9bdfd66de6d" containerName="maas-api" Nov 03 15:48:38.848665 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.848447 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37f9ec1-b784-41fd-9a79-8df424aa77e5" containerName="istio-proxy" Nov 03 15:48:38.851513 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.851497 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc" Nov 03 15:48:38.911884 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.911828 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc"] Nov 03 15:48:38.954204 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.954180 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:48:38.954340 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.954214 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 15:48:38.967442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.967416 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176c6b76-8cef-485d-8e85-22ca3f443121-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f77c3462-q5dmc\" (UID: \"176c6b76-8cef-485d-8e85-22ca3f443121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc" Nov 03 15:48:38.967564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.967483 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nkfw\" (UniqueName: \"kubernetes.io/projected/176c6b76-8cef-485d-8e85-22ca3f443121-kube-api-access-9nkfw\") pod \"maas-api-revoke-sub-f77c3462-q5dmc\" (UID: \"176c6b76-8cef-485d-8e85-22ca3f443121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc" Nov 03 15:48:38.967564 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:38.967545 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176c6b76-8cef-485d-8e85-22ca3f443121-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f77c3462-q5dmc\" (UID: \"176c6b76-8cef-485d-8e85-22ca3f443121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc" Nov 03 15:48:39.068296 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:39.068259 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176c6b76-8cef-485d-8e85-22ca3f443121-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f77c3462-q5dmc\" (UID: \"176c6b76-8cef-485d-8e85-22ca3f443121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc" Nov 03 15:48:39.068502 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:39.068326 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nkfw\" (UniqueName: \"kubernetes.io/projected/176c6b76-8cef-485d-8e85-22ca3f443121-kube-api-access-9nkfw\") pod \"maas-api-revoke-sub-f77c3462-q5dmc\" (UID: \"176c6b76-8cef-485d-8e85-22ca3f443121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc" Nov 03 15:48:39.068502 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:39.068349 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176c6b76-8cef-485d-8e85-22ca3f443121-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f77c3462-q5dmc\" (UID: \"176c6b76-8cef-485d-8e85-22ca3f443121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc" Nov 03 15:48:39.068945 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:39.068924 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176c6b76-8cef-485d-8e85-22ca3f443121-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f77c3462-q5dmc\" (UID: \"176c6b76-8cef-485d-8e85-22ca3f443121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc" Nov 03 15:48:39.070592 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:39.070559 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176c6b76-8cef-485d-8e85-22ca3f443121-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f77c3462-q5dmc\" (UID: \"176c6b76-8cef-485d-8e85-22ca3f443121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc" Nov 03 15:48:39.129489 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:39.129429 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nkfw\" (UniqueName: \"kubernetes.io/projected/176c6b76-8cef-485d-8e85-22ca3f443121-kube-api-access-9nkfw\") pod \"maas-api-revoke-sub-f77c3462-q5dmc\" (UID: \"176c6b76-8cef-485d-8e85-22ca3f443121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc" Nov 03 15:48:39.163496 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:39.163475 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc" Nov 03 15:48:39.549317 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:39.549180 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc"] Nov 03 15:48:39.551690 ip-10-0-139-207 kubenswrapper[4557]: W1103 15:48:39.551666 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176c6b76_8cef_485d_8e85_22ca3f443121.slice/crio-8c2182ce4ca26789eefd69d1451509e31cca609f99388ff6d438ca15e6f978ab WatchSource:0}: Error finding container 8c2182ce4ca26789eefd69d1451509e31cca609f99388ff6d438ca15e6f978ab: Status 404 returned error can't find the container with id 8c2182ce4ca26789eefd69d1451509e31cca609f99388ff6d438ca15e6f978ab Nov 03 15:48:40.263985 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.263885 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" containerID="cri-o://9dea0f430fee7361fdf2ad4457d4f1670dd7ffa674fa0c6f81229e423fb0038a" gracePeriod=45 Nov 03 15:48:40.528784 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.528727 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:48:40.554962 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.554934 4557 generic.go:356] "Generic (PLEG): container finished" podID="5256df6b-b047-4aae-82da-fba43402fbe0" containerID="9dea0f430fee7361fdf2ad4457d4f1670dd7ffa674fa0c6f81229e423fb0038a" exitCode=0 Nov 03 15:48:40.555093 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.554981 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" Nov 03 15:48:40.555093 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.555014 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" event={"ID":"5256df6b-b047-4aae-82da-fba43402fbe0","Type":"ContainerDied","Data":"9dea0f430fee7361fdf2ad4457d4f1670dd7ffa674fa0c6f81229e423fb0038a"} Nov 03 15:48:40.555093 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.555054 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg" event={"ID":"5256df6b-b047-4aae-82da-fba43402fbe0","Type":"ContainerDied","Data":"9f298760d5491b2614bfcc84d3acb8a215f1f00527e8d091fc06c2ca134ae0d1"} Nov 03 15:48:40.555093 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.555076 4557 scope.go:122] "RemoveContainer" containerID="9dea0f430fee7361fdf2ad4457d4f1670dd7ffa674fa0c6f81229e423fb0038a" Nov 03 15:48:40.556543 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.556520 4557 generic.go:356] "Generic (PLEG): container finished" podID="176c6b76-8cef-485d-8e85-22ca3f443121" containerID="5b805d47908fe34742c64f12987e99aa5175a5d0e9d3e3c9ac72d48de714a276" exitCode=0 Nov 03 15:48:40.556643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.556550 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc" event={"ID":"176c6b76-8cef-485d-8e85-22ca3f443121","Type":"ContainerDied","Data":"5b805d47908fe34742c64f12987e99aa5175a5d0e9d3e3c9ac72d48de714a276"} Nov 03 15:48:40.556643 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.556624 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc" event={"ID":"176c6b76-8cef-485d-8e85-22ca3f443121","Type":"ContainerStarted","Data":"8c2182ce4ca26789eefd69d1451509e31cca609f99388ff6d438ca15e6f978ab"} Nov 03 15:48:40.563245 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.563206 4557 scope.go:122] "RemoveContainer" containerID="9dea0f430fee7361fdf2ad4457d4f1670dd7ffa674fa0c6f81229e423fb0038a" Nov 03 15:48:40.563509 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:48:40.563491 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dea0f430fee7361fdf2ad4457d4f1670dd7ffa674fa0c6f81229e423fb0038a\": container with ID starting with 9dea0f430fee7361fdf2ad4457d4f1670dd7ffa674fa0c6f81229e423fb0038a not found: ID does not exist" containerID="9dea0f430fee7361fdf2ad4457d4f1670dd7ffa674fa0c6f81229e423fb0038a" Nov 03 15:48:40.563560 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.563519 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dea0f430fee7361fdf2ad4457d4f1670dd7ffa674fa0c6f81229e423fb0038a"} err="failed to get container status \"9dea0f430fee7361fdf2ad4457d4f1670dd7ffa674fa0c6f81229e423fb0038a\": rpc error: code = NotFound desc = could not find container \"9dea0f430fee7361fdf2ad4457d4f1670dd7ffa674fa0c6f81229e423fb0038a\": container with ID starting with 9dea0f430fee7361fdf2ad4457d4f1670dd7ffa674fa0c6f81229e423fb0038a not found: ID does not exist" Nov 03 15:48:40.683322 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.683293 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-model-cache\") pod \"5256df6b-b047-4aae-82da-fba43402fbe0\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " Nov 03 15:48:40.683493 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.683358 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5256df6b-b047-4aae-82da-fba43402fbe0-tls-certs\" (UniqueName: \"kubernetes.io/secret/5256df6b-b047-4aae-82da-fba43402fbe0-tls-certs\") pod \"5256df6b-b047-4aae-82da-fba43402fbe0\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " Nov 03 15:48:40.683493 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.683392 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-home\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-home\") pod \"5256df6b-b047-4aae-82da-fba43402fbe0\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " Nov 03 15:48:40.683493 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.683432 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5256df6b-b047-4aae-82da-fba43402fbe0-kube-api-access-7d7dg\" (UniqueName: \"kubernetes.io/projected/5256df6b-b047-4aae-82da-fba43402fbe0-kube-api-access-7d7dg\") pod \"5256df6b-b047-4aae-82da-fba43402fbe0\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " Nov 03 15:48:40.683493 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.683474 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-dshm\") pod \"5256df6b-b047-4aae-82da-fba43402fbe0\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " Nov 03 15:48:40.683780 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.683503 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-tmp-dir\") pod \"5256df6b-b047-4aae-82da-fba43402fbe0\" (UID: \"5256df6b-b047-4aae-82da-fba43402fbe0\") " Nov 03 15:48:40.683780 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.683597 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-model-cache" pod "5256df6b-b047-4aae-82da-fba43402fbe0" (UID: "5256df6b-b047-4aae-82da-fba43402fbe0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:48:40.683926 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.683788 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-home" pod "5256df6b-b047-4aae-82da-fba43402fbe0" (UID: "5256df6b-b047-4aae-82da-fba43402fbe0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:48:40.683999 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.683936 4557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-model-cache\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:40.683999 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.683956 4557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-home\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:40.684117 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.684065 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-tmp-dir" pod "5256df6b-b047-4aae-82da-fba43402fbe0" (UID: "5256df6b-b047-4aae-82da-fba43402fbe0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:48:40.685777 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.685749 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5256df6b-b047-4aae-82da-fba43402fbe0-tls-certs" pod "5256df6b-b047-4aae-82da-fba43402fbe0" (UID: "5256df6b-b047-4aae-82da-fba43402fbe0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:48:40.685777 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.685755 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-dshm" pod "5256df6b-b047-4aae-82da-fba43402fbe0" (UID: "5256df6b-b047-4aae-82da-fba43402fbe0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:48:40.685940 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.685885 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5256df6b-b047-4aae-82da-fba43402fbe0-kube-api-access-7d7dg" pod "5256df6b-b047-4aae-82da-fba43402fbe0" (UID: "5256df6b-b047-4aae-82da-fba43402fbe0"). InnerVolumeSpecName "kube-api-access-7d7dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:48:40.785251 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.785186 4557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-dshm\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:40.785251 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.785230 4557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5256df6b-b047-4aae-82da-fba43402fbe0-tmp-dir\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:40.785251 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.785242 4557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5256df6b-b047-4aae-82da-fba43402fbe0-tls-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:40.785496 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.785256 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d7dg\" (UniqueName: \"kubernetes.io/projected/5256df6b-b047-4aae-82da-fba43402fbe0-kube-api-access-7d7dg\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:40.975257 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.975222 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg"] Nov 03 15:48:40.983267 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:40.983242 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/distinct-2-w3-8b841a-kserve-76685757b9-fzrlg"] Nov 03 15:48:41.694991 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:41.694965 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc" Nov 03 15:48:41.794552 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:41.794530 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176c6b76-8cef-485d-8e85-22ca3f443121-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176c6b76-8cef-485d-8e85-22ca3f443121-maas-api-xks-ca\") pod \"176c6b76-8cef-485d-8e85-22ca3f443121\" (UID: \"176c6b76-8cef-485d-8e85-22ca3f443121\") " Nov 03 15:48:41.794709 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:41.794601 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/176c6b76-8cef-485d-8e85-22ca3f443121-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176c6b76-8cef-485d-8e85-22ca3f443121-maas-api-openshift-ca\") pod \"176c6b76-8cef-485d-8e85-22ca3f443121\" (UID: \"176c6b76-8cef-485d-8e85-22ca3f443121\") " Nov 03 15:48:41.794780 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:41.794705 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176c6b76-8cef-485d-8e85-22ca3f443121-kube-api-access-9nkfw\" (UniqueName: \"kubernetes.io/projected/176c6b76-8cef-485d-8e85-22ca3f443121-kube-api-access-9nkfw\") pod \"176c6b76-8cef-485d-8e85-22ca3f443121\" (UID: \"176c6b76-8cef-485d-8e85-22ca3f443121\") " Nov 03 15:48:41.794927 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:41.794903 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176c6b76-8cef-485d-8e85-22ca3f443121-maas-api-openshift-ca" pod "176c6b76-8cef-485d-8e85-22ca3f443121" (UID: "176c6b76-8cef-485d-8e85-22ca3f443121"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:48:41.796497 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:41.796473 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176c6b76-8cef-485d-8e85-22ca3f443121-maas-api-xks-ca" pod "176c6b76-8cef-485d-8e85-22ca3f443121" (UID: "176c6b76-8cef-485d-8e85-22ca3f443121"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:48:41.796613 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:41.796574 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176c6b76-8cef-485d-8e85-22ca3f443121-kube-api-access-9nkfw" pod "176c6b76-8cef-485d-8e85-22ca3f443121" (UID: "176c6b76-8cef-485d-8e85-22ca3f443121"). InnerVolumeSpecName "kube-api-access-9nkfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:48:41.895987 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:41.895913 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nkfw\" (UniqueName: \"kubernetes.io/projected/176c6b76-8cef-485d-8e85-22ca3f443121-kube-api-access-9nkfw\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:41.895987 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:41.895953 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176c6b76-8cef-485d-8e85-22ca3f443121-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:41.895987 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:41.895963 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176c6b76-8cef-485d-8e85-22ca3f443121-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:48:42.283860 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:42.283779 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" path="/var/lib/kubelet/pods/5256df6b-b047-4aae-82da-fba43402fbe0/volumes" Nov 03 15:48:42.568517 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:42.568484 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc" event={"ID":"176c6b76-8cef-485d-8e85-22ca3f443121","Type":"ContainerDied","Data":"8c2182ce4ca26789eefd69d1451509e31cca609f99388ff6d438ca15e6f978ab"} Nov 03 15:48:42.568717 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:42.568537 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c2182ce4ca26789eefd69d1451509e31cca609f99388ff6d438ca15e6f978ab" Nov 03 15:48:42.568717 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:42.568495 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc" Nov 03 15:48:46.529011 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:48:46.528974 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:48:54.309727 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:54.309693 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:48:55.893661 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:55.893627 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:48:56.190144 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:56.190027 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:48:57.055805 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:48:57.055764 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:49:03.404870 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:03.404814 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:49:03.638046 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:03.638000 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:49:04.784071 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:04.784034 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:49:05.608983 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:05.608949 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:49:05.899617 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:05.899528 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:49:05.974964 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:05.974925 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl"] Nov 03 15:49:05.975157 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:05.975137 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" podUID="a4ea4afc-4c65-49fb-a234-b1d695a8a868" containerName="istio-proxy" containerID="cri-o://6279aff283b48494bc0cc0a6142299ab44fd9faa0fefdcc04157a01f0d1390ba" gracePeriod=30 Nov 03 15:49:06.116309 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:06.116279 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:49:08.497501 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:08.497467 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:49:09.828591 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:09.828549 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:49:11.226421 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.226392 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:49:11.384887 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.384857 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-data\") pod \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " Nov 03 15:49:11.385052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.384895 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-workload-socket\") pod \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " Nov 03 15:49:11.385052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.384929 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-podinfo\") pod \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " Nov 03 15:49:11.385052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.384971 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-token\" (UniqueName: \"kubernetes.io/projected/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-token\") pod \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " Nov 03 15:49:11.385052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.384994 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-workload-certs\") pod \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " Nov 03 15:49:11.385052 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.385021 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-credential-socket\") pod \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " Nov 03 15:49:11.385330 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.385103 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ea4afc-4c65-49fb-a234-b1d695a8a868-kube-api-access-rr5tn\" (UniqueName: \"kubernetes.io/projected/a4ea4afc-4c65-49fb-a234-b1d695a8a868-kube-api-access-rr5tn\") pod \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " Nov 03 15:49:11.385330 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.385145 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-envoy\") pod \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " Nov 03 15:49:11.385330 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.385178 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istiod-ca-cert\") pod \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\" (UID: \"a4ea4afc-4c65-49fb-a234-b1d695a8a868\") " Nov 03 15:49:11.385492 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.385172 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-data" pod "a4ea4afc-4c65-49fb-a234-b1d695a8a868" (UID: "a4ea4afc-4c65-49fb-a234-b1d695a8a868"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:49:11.385492 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.385414 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-credential-socket" pod "a4ea4afc-4c65-49fb-a234-b1d695a8a868" (UID: "a4ea4afc-4c65-49fb-a234-b1d695a8a868"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:49:11.385492 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.385183 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-workload-socket" pod "a4ea4afc-4c65-49fb-a234-b1d695a8a868" (UID: "a4ea4afc-4c65-49fb-a234-b1d695a8a868"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:49:11.385492 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.385309 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-workload-certs" pod "a4ea4afc-4c65-49fb-a234-b1d695a8a868" (UID: "a4ea4afc-4c65-49fb-a234-b1d695a8a868"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:49:11.385699 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.385562 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istiod-ca-cert" pod "a4ea4afc-4c65-49fb-a234-b1d695a8a868" (UID: "a4ea4afc-4c65-49fb-a234-b1d695a8a868"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:49:11.385699 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.385566 4557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-workload-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.385699 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.385643 4557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-credential-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.385699 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.385664 4557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-data\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.385699 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.385681 4557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-workload-socket\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.387231 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.387209 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ea4afc-4c65-49fb-a234-b1d695a8a868-kube-api-access-rr5tn" pod "a4ea4afc-4c65-49fb-a234-b1d695a8a868" (UID: "a4ea4afc-4c65-49fb-a234-b1d695a8a868"). InnerVolumeSpecName "kube-api-access-rr5tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:49:11.387558 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.387519 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-podinfo" pod "a4ea4afc-4c65-49fb-a234-b1d695a8a868" (UID: "a4ea4afc-4c65-49fb-a234-b1d695a8a868"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:49:11.387698 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.387618 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-token" pod "a4ea4afc-4c65-49fb-a234-b1d695a8a868" (UID: "a4ea4afc-4c65-49fb-a234-b1d695a8a868"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:49:11.387698 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.387632 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-envoy" pod "a4ea4afc-4c65-49fb-a234-b1d695a8a868" (UID: "a4ea4afc-4c65-49fb-a234-b1d695a8a868"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:49:11.486705 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.486662 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr5tn\" (UniqueName: \"kubernetes.io/projected/a4ea4afc-4c65-49fb-a234-b1d695a8a868-kube-api-access-rr5tn\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.486705 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.486698 4557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-envoy\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.486705 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.486713 4557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istiod-ca-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.487012 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.486728 4557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-podinfo\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.487012 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.486741 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ea4afc-4c65-49fb-a234-b1d695a8a868-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:49:11.710116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.710027 4557 generic.go:356] "Generic (PLEG): container finished" podID="a4ea4afc-4c65-49fb-a234-b1d695a8a868" containerID="6279aff283b48494bc0cc0a6142299ab44fd9faa0fefdcc04157a01f0d1390ba" exitCode=0 Nov 03 15:49:11.710116 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.710079 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" Nov 03 15:49:11.710294 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.710111 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" event={"ID":"a4ea4afc-4c65-49fb-a234-b1d695a8a868","Type":"ContainerDied","Data":"6279aff283b48494bc0cc0a6142299ab44fd9faa0fefdcc04157a01f0d1390ba"} Nov 03 15:49:11.710294 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.710165 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl" event={"ID":"a4ea4afc-4c65-49fb-a234-b1d695a8a868","Type":"ContainerDied","Data":"b3c660ac4a20e3ab2ecf0f4abdd7be70ffbb87f6bb6e06353376560cf6ff88d1"} Nov 03 15:49:11.710294 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.710188 4557 scope.go:122] "RemoveContainer" containerID="6279aff283b48494bc0cc0a6142299ab44fd9faa0fefdcc04157a01f0d1390ba" Nov 03 15:49:11.719347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.719333 4557 scope.go:122] "RemoveContainer" containerID="6279aff283b48494bc0cc0a6142299ab44fd9faa0fefdcc04157a01f0d1390ba" Nov 03 15:49:11.719606 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:49:11.719571 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6279aff283b48494bc0cc0a6142299ab44fd9faa0fefdcc04157a01f0d1390ba\": container with ID starting with 6279aff283b48494bc0cc0a6142299ab44fd9faa0fefdcc04157a01f0d1390ba not found: ID does not exist" containerID="6279aff283b48494bc0cc0a6142299ab44fd9faa0fefdcc04157a01f0d1390ba" Nov 03 15:49:11.719655 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.719615 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6279aff283b48494bc0cc0a6142299ab44fd9faa0fefdcc04157a01f0d1390ba"} err="failed to get container status \"6279aff283b48494bc0cc0a6142299ab44fd9faa0fefdcc04157a01f0d1390ba\": rpc error: code = NotFound desc = could not find container \"6279aff283b48494bc0cc0a6142299ab44fd9faa0fefdcc04157a01f0d1390ba\": container with ID starting with 6279aff283b48494bc0cc0a6142299ab44fd9faa0fefdcc04157a01f0d1390ba not found: ID does not exist" Nov 03 15:49:11.786278 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.786233 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:49:11.886681 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.886648 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl"] Nov 03 15:49:11.937130 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:11.937080 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-pj9zl"] Nov 03 15:49:12.283455 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:12.283419 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4ea4afc-4c65-49fb-a234-b1d695a8a868" path="/var/lib/kubelet/pods/a4ea4afc-4c65-49fb-a234-b1d695a8a868/volumes" Nov 03 15:49:16.533360 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:49:16.533319 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:49:46.529282 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:49:46.529204 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:49:49.768454 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:49.768420 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:49:52.449729 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:49:52.449691 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:50:10.142928 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:50:10.142891 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:50:10.569041 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:50:10.569009 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:50:16.529383 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:50:16.529347 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:50:21.722462 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:50:21.722429 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:50:28.306621 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:50:28.306575 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:50:31.457864 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:50:31.457831 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:50:40.463788 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:50:40.463745 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5952j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:50:40.464187 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:50:40.463822 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" podUID="6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:50:46.528449 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:50:46.528407 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:51:10.499882 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:10.499840 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:51:16.529556 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:51:16.529517 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:51:17.087439 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:17.087408 4557 scope.go:122] "RemoveContainer" containerID="bffb3e6fd991a0ba63f82d32cf32bd30ff5ceb11dce73ef3d42d9f02b2d166c3" Nov 03 15:51:21.415767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:21.415730 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:51:25.435471 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:25.435443 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:51:25.626749 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:25.626715 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:51:28.779589 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:28.779539 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8"] Nov 03 15:51:28.780000 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:28.779809 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" podUID="fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e" containerName="discovery" containerID="cri-o://84919aa5aaee1f580058d126cd4e970b344c99184010c22b6ac436b9a84426fc" gracePeriod=30 Nov 03 15:51:29.030062 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.030002 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:51:29.134381 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.134355 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-cacerts\" (UniqueName: \"kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-cacerts\") pod \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " Nov 03 15:51:29.134563 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.134393 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-kube-api-access-62glw\" (UniqueName: \"kubernetes.io/projected/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-kube-api-access-62glw\") pod \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " Nov 03 15:51:29.134563 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.134425 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-csr-ca-configmap\") pod \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " Nov 03 15:51:29.134563 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.134471 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-kubeconfig\") pod \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " Nov 03 15:51:29.134563 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.134504 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-local-certs\") pod \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " Nov 03 15:51:29.134563 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.134553 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-token\" (UniqueName: \"kubernetes.io/projected/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-token\") pod \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " Nov 03 15:51:29.134878 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.134634 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-csr-dns-cert\") pod \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\" (UID: \"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e\") " Nov 03 15:51:29.135138 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.135091 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-csr-ca-configmap" pod "fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e" (UID: "fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:51:29.137209 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.137177 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-cacerts" pod "fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e" (UID: "fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:51:29.137389 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.137364 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-csr-dns-cert" pod "fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e" (UID: "fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:51:29.137510 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.137456 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-token" pod "fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e" (UID: "fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:51:29.137654 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.137633 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-kube-api-access-62glw" pod "fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e" (UID: "fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e"). InnerVolumeSpecName "kube-api-access-62glw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:51:29.137717 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.137696 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-kubeconfig" pod "fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e" (UID: "fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:51:29.137767 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.137748 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-local-certs" pod "fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e" (UID: "fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:51:29.236210 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.236183 4557 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-cacerts\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:51:29.236210 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.236210 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62glw\" (UniqueName: \"kubernetes.io/projected/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-kube-api-access-62glw\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:51:29.236417 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.236222 4557 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-csr-ca-configmap\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:51:29.236417 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.236233 4557 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-kubeconfig\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:51:29.236417 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.236242 4557 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-local-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:51:29.236417 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.236250 4557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-token\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:51:29.236417 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.236260 4557 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e-istio-csr-dns-cert\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 15:51:29.438225 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.438192 4557 generic.go:356] "Generic (PLEG): container finished" podID="fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e" containerID="84919aa5aaee1f580058d126cd4e970b344c99184010c22b6ac436b9a84426fc" exitCode=0 Nov 03 15:51:29.438407 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.438241 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" Nov 03 15:51:29.438407 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.438274 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" event={"ID":"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e","Type":"ContainerDied","Data":"84919aa5aaee1f580058d126cd4e970b344c99184010c22b6ac436b9a84426fc"} Nov 03 15:51:29.438407 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.438330 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8" event={"ID":"fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e","Type":"ContainerDied","Data":"96bbc3596340270883aaa783bbd09cb33f0b72fb596f76d8d901c85b5fa643a2"} Nov 03 15:51:29.438407 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.438351 4557 scope.go:122] "RemoveContainer" containerID="84919aa5aaee1f580058d126cd4e970b344c99184010c22b6ac436b9a84426fc" Nov 03 15:51:29.451035 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.451012 4557 scope.go:122] "RemoveContainer" containerID="84919aa5aaee1f580058d126cd4e970b344c99184010c22b6ac436b9a84426fc" Nov 03 15:51:29.451314 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:51:29.451296 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84919aa5aaee1f580058d126cd4e970b344c99184010c22b6ac436b9a84426fc\": container with ID starting with 84919aa5aaee1f580058d126cd4e970b344c99184010c22b6ac436b9a84426fc not found: ID does not exist" containerID="84919aa5aaee1f580058d126cd4e970b344c99184010c22b6ac436b9a84426fc" Nov 03 15:51:29.451385 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.451327 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84919aa5aaee1f580058d126cd4e970b344c99184010c22b6ac436b9a84426fc"} err="failed to get container status \"84919aa5aaee1f580058d126cd4e970b344c99184010c22b6ac436b9a84426fc\": rpc error: code = NotFound desc = could not find container \"84919aa5aaee1f580058d126cd4e970b344c99184010c22b6ac436b9a84426fc\": container with ID starting with 84919aa5aaee1f580058d126cd4e970b344c99184010c22b6ac436b9a84426fc not found: ID does not exist" Nov 03 15:51:29.604434 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.604405 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8"] Nov 03 15:51:29.654533 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:29.654505 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2scp8"] Nov 03 15:51:30.290004 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:30.289954 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e" path="/var/lib/kubelet/pods/fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e/volumes" Nov 03 15:51:36.049653 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:36.049615 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt"] Nov 03 15:51:36.099299 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:36.099260 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f42a5c8-cl4jt"] Nov 03 15:51:36.283686 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:36.283644 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="320d7c16-9625-45ed-bd30-56c0eaf2bbf9" path="/var/lib/kubelet/pods/320d7c16-9625-45ed-bd30-56c0eaf2bbf9/volumes" Nov 03 15:51:46.529989 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:51:46.529950 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:51:54.064657 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:54.064621 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb"] Nov 03 15:51:54.114347 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:54.114319 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb2511a3-hbdfb"] Nov 03 15:51:54.283646 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:54.283610 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b96a745-452f-4a97-b177-abe512337570" path="/var/lib/kubelet/pods/0b96a745-452f-4a97-b177-abe512337570/volumes" Nov 03 15:51:55.049130 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:55.049095 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6"] Nov 03 15:51:55.099107 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:55.099074 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab9a0da-j27d6"] Nov 03 15:51:56.283309 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:51:56.283276 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b16ddb97-4e08-46ba-bf10-8a24dbd02b26" path="/var/lib/kubelet/pods/b16ddb97-4e08-46ba-bf10-8a24dbd02b26/volumes" Nov 03 15:52:08.049607 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:52:08.049558 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx"] Nov 03 15:52:08.099748 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:52:08.099712 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bd373bc-ktlbx"] Nov 03 15:52:08.284515 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:52:08.284478 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7bd7224-6371-4352-a5ef-37bcf2155cf7" path="/var/lib/kubelet/pods/b7bd7224-6371-4352-a5ef-37bcf2155cf7/volumes" Nov 03 15:52:10.871894 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:52:10.871858 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:52:16.531462 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:52:16.531421 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:52:17.106880 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:52:17.106845 4557 scope.go:122] "RemoveContainer" containerID="b042d841744b12df381f55f1347438ef8d61bf40b9f2d2231186bdba0b189354" Nov 03 15:52:17.114900 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:52:17.114882 4557 scope.go:122] "RemoveContainer" containerID="75446c073bde33da85165e6f6c0f560a3d0ad5eeaf20e3065acf5a039c3b4cc2" Nov 03 15:52:46.528401 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:52:46.528313 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:53:09.630402 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:53:09.630363 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:53:09.825783 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:53:09.825753 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:53:10.620862 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:53:10.620837 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:53:10.628900 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:53:10.628881 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:53:11.218759 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:53:11.218724 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:53:11.623253 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:53:11.623211 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:53:16.528526 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:53:16.528489 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:53:17.140426 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:53:17.140395 4557 scope.go:122] "RemoveContainer" containerID="29c94fa6ead7b19ad438192ab77e89354b285b0cf9b9d0a67c5ffcb3f5fcfb1f" Nov 03 15:53:17.148419 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:53:17.148400 4557 scope.go:122] "RemoveContainer" containerID="4f6ac07c9dd0c4ee551e374fc64eb56811b7482b166fa4e4a63f2369297cd7f9" Nov 03 15:53:17.159726 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:53:17.159705 4557 scope.go:122] "RemoveContainer" containerID="4e032cb8127b1e9c5f0bd9ed07c45ad08d245ce07fe27a858c59840363675727" Nov 03 15:53:17.167316 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:53:17.167288 4557 scope.go:122] "RemoveContainer" containerID="a38690967e9f9ff121f9254ee4f17b404658f9bec5ca5d31be4e1e4f5e337f90" Nov 03 15:53:17.175736 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:53:17.175718 4557 scope.go:122] "RemoveContainer" containerID="071e23befcc092ef8e1c669a76b5aee215ab69fcdc23f60d7928bdb4aadd664c" Nov 03 15:53:17.184310 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:53:17.184296 4557 scope.go:122] "RemoveContainer" containerID="b131eeba5fcab0e7b86e6788896486450f6b9c167b4f178d866ce3ea6318e975" Nov 03 15:53:17.191060 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:53:17.191042 4557 scope.go:122] "RemoveContainer" containerID="3c11b3d3b836e0657be7ef2b72ca4db0d28b222e06a3d1c8eea6939aa76583e3" Nov 03 15:53:25.454461 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:53:25.454405 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:53:25.640906 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:53:25.640857 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:53:42.047996 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:53:42.047966 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc"] Nov 03 15:53:42.097892 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:53:42.097863 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77c3462-q5dmc"] Nov 03 15:53:42.283901 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:53:42.283865 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176c6b76-8cef-485d-8e85-22ca3f443121" path="/var/lib/kubelet/pods/176c6b76-8cef-485d-8e85-22ca3f443121/volumes" Nov 03 15:53:46.529446 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:53:46.529398 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:54:10.732483 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:54:10.732437 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:54:11.667435 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:54:11.667403 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:54:16.529222 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:54:16.529183 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:54:46.567314 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:54:46.567277 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:55:12.036753 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:55:12.036694 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:55:14.277095 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:55:14.277062 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:55:16.528694 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:55:16.528659 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:55:17.232511 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:55:17.232479 4557 scope.go:122] "RemoveContainer" containerID="5b805d47908fe34742c64f12987e99aa5175a5d0e9d3e3c9ac72d48de714a276" Nov 03 15:55:46.528575 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:55:46.528499 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:56:12.393746 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:56:12.393701 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:56:16.528594 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:56:16.528537 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:56:46.528320 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:56:46.528285 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:56:50.991655 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:56:50.991623 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:56:51.702939 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:56:51.702905 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:56:51.903135 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:56:51.903081 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:56:56.093118 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:56:56.092893 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:56:56.360806 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:56:56.360721 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:56:59.340713 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:56:59.340671 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:56:59.534300 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:56:59.534266 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:02.266663 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:02.266633 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:02.464355 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:02.464324 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:07.294371 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:07.294332 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:12.324390 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:12.324356 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:12.740482 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:12.740374 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:16.528126 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:57:16.528086 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:57:17.267335 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:17.267291 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:21.001203 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:21.001168 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:21.630207 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:21.630172 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:21.835442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:21.835396 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:22.513489 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:22.513457 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:25.316438 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:25.316402 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:27.422553 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:27.422509 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:32.307810 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:32.307765 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:37.214733 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:37.214690 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:42.105257 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:42.105222 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:46.528960 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:57:46.528925 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:57:46.932178 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:46.932124 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:47.133446 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:47.133415 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:51.866710 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:51.866675 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:57:56.752006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:57:56.751947 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:01.620238 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:01.620193 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:06.497455 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:06.497397 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:10.647725 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:10.647693 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:58:10.655475 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:10.655451 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 15:58:11.418390 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:11.418310 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:13.096678 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:13.096648 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:13.497393 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:13.497298 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:16.338359 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:16.338320 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:16.529094 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:58:16.529057 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:58:21.258765 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:21.258718 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:21.455006 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:21.454973 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:26.415487 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:26.415385 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:26.714765 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:26.714670 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:31.696752 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:31.696721 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:33.799835 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:33.799790 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:34.025158 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:34.025115 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:36.427543 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:36.427511 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:40.795080 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:40.795048 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:42.311918 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:42.311880 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:42.584208 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:42.584105 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:43.389275 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:43.389241 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:46.529413 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:58:46.529376 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:58:49.826306 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:49.826273 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:50.051442 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:50.051404 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:58:50.451936 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:58:50.451903 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:59:04.281638 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:59:04.281606 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:59:04.892417 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:59:04.892382 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:59:10.919424 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:59:10.919369 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:59:13.477903 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:59:13.477871 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:59:16.528707 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:59:16.528671 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:59:25.354684 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:59:25.354651 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:59:46.528641 ip-10-0-139-207 kubenswrapper[4557]: E1103 15:59:46.528601 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 15:59:48.602400 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:59:48.602361 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 15:59:48.812378 ip-10-0-139-207 kubenswrapper[4557]: I1103 15:59:48.812341 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:00:13.808393 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:00:13.808357 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:00:16.529148 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:00:16.529108 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:00:46.529783 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:00:46.529741 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:00:55.464725 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:00:55.464690 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5952j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:00:55.465137 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:00:55.464747 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5952j" podUID="6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:01:14.175598 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:01:14.175506 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:01:15.950654 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:01:15.950608 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:01:16.569783 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:01:16.569748 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:01:38.770680 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:01:38.770597 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:01:39.485477 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:01:39.485439 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:01:39.741141 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:01:39.741061 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:01:43.692217 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:01:43.692179 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:01:43.907574 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:01:43.907536 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:01:44.158101 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:01:44.158069 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:01:46.528962 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:01:46.528919 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:01:46.996811 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:01:46.996776 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:01:47.287369 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:01:47.287285 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:01:50.153677 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:01:50.153643 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:01:50.350900 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:01:50.350871 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:01:50.540833 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:01:50.540754 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:01:55.214339 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:01:55.214310 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:00.107922 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:00.107882 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:04.942886 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:04.942849 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:08.551604 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:08.551547 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:09.159300 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:09.159272 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:09.352377 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:09.352344 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:10.197487 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:10.197444 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:12.567942 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:12.567905 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:14.537635 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:14.537598 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:15.020114 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:15.020080 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:16.528139 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:02:16.528104 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:02:17.784815 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:17.784782 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:17.981537 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:17.981505 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:20.000735 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:20.000694 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:24.897864 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:24.897829 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:29.816324 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:29.816292 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:34.715245 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:34.715209 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:39.628842 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:39.628790 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:44.513529 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:44.513495 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:46.528764 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:02:46.528724 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:02:49.437710 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:49.437677 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:54.305066 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:54.304981 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:58.269763 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:58.269735 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:58.459635 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:58.459601 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:02:59.212349 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:02:59.212318 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:04.112418 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:04.112381 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:08.674883 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:08.674852 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:09.043389 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:09.043292 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:10.674250 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:10.674222 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 16:03:10.682417 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:10.682398 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 16:03:13.719614 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:13.719557 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:14.333559 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:14.333528 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:14.880209 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:14.880179 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:16.529318 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:03:16.529282 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:03:19.219414 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:19.219372 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:21.047812 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:21.047767 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:21.237717 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:21.237684 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:24.018757 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:24.018693 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:24.640097 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:24.640061 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:28.409006 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:28.408961 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:29.397234 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:29.397199 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:29.857731 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:29.857694 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:30.053681 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:30.053651 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:30.243690 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:30.243611 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:30.919760 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:30.919722 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:32.029694 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:32.029658 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:37.030232 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:37.030198 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:37.222107 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:37.222073 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:37.439189 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:37.439148 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:38.444635 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:38.444605 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:03:46.528640 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:03:46.528603 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:03:50.812536 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:03:50.812505 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:04:04.789764 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:04:04.789729 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:04:04.979914 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:04:04.979881 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:04:14.461267 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:04:14.461233 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:04:15.233249 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:04:15.233210 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:04:15.627761 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:04:15.627725 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:04:16.528784 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:04:16.528748 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:04:46.529148 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:04:46.529071 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:04:49.846610 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:04:49.843426 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:04:58.918689 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:04:58.918660 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:05:07.774454 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:07.774416 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:05:11.899321 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.899284 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv"] Nov 03 16:05:11.912603 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.900367 4557 cpu_manager.go:414] "containerMap: removing container" podUID="74678262-2f6e-41a6-aa2b-00b64aa2c545" containerName="istio-proxy" Nov 03 16:05:11.912809 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.912788 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74678262-2f6e-41a6-aa2b-00b64aa2c545" containerName="istio-proxy" Nov 03 16:05:11.912925 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.912912 4557 cpu_manager.go:414] "containerMap: removing container" podUID="f7bb3b78-db67-4de5-9b60-10d26274a199" containerName="istio-proxy" Nov 03 16:05:11.913020 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.913008 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7bb3b78-db67-4de5-9b60-10d26274a199" containerName="istio-proxy" Nov 03 16:05:11.913123 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.913113 4557 cpu_manager.go:414] "containerMap: removing container" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" Nov 03 16:05:11.913211 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.913201 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" Nov 03 16:05:11.913303 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.913293 4557 cpu_manager.go:414] "containerMap: removing container" podUID="176c6b76-8cef-485d-8e85-22ca3f443121" containerName="revoke-keys" Nov 03 16:05:11.913383 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.913372 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176c6b76-8cef-485d-8e85-22ca3f443121" containerName="revoke-keys" Nov 03 16:05:11.913470 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.913460 4557 cpu_manager.go:414] "containerMap: removing container" podUID="a4ea4afc-4c65-49fb-a234-b1d695a8a868" containerName="istio-proxy" Nov 03 16:05:11.913551 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.913541 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4ea4afc-4c65-49fb-a234-b1d695a8a868" containerName="istio-proxy" Nov 03 16:05:11.913660 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.913646 4557 cpu_manager.go:414] "containerMap: removing container" podUID="fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e" containerName="discovery" Nov 03 16:05:11.913751 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.913740 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e" containerName="discovery" Nov 03 16:05:11.913953 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.913936 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4ea4afc-4c65-49fb-a234-b1d695a8a868" containerName="istio-proxy" Nov 03 16:05:11.914058 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.914046 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa511cdc-f93b-4ec6-b490-f11eb1aa5b0e" containerName="discovery" Nov 03 16:05:11.914154 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.914143 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="74678262-2f6e-41a6-aa2b-00b64aa2c545" containerName="istio-proxy" Nov 03 16:05:11.914248 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.914238 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="5256df6b-b047-4aae-82da-fba43402fbe0" containerName="main" Nov 03 16:05:11.914338 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.914328 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="176c6b76-8cef-485d-8e85-22ca3f443121" containerName="revoke-keys" Nov 03 16:05:11.926604 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.920900 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv" Nov 03 16:05:11.963499 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:11.963471 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv"] Nov 03 16:05:12.012033 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:12.011843 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:05:12.012033 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:12.012002 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\"" Nov 03 16:05:12.018334 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:12.018306 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmjfw\" (UniqueName: \"kubernetes.io/projected/42dc28cf-a10c-43ae-b868-4d4a79bab217-kube-api-access-pmjfw\") pod \"maas-api-revoke-sub-3e31a506-fhfnv\" (UID: \"42dc28cf-a10c-43ae-b868-4d4a79bab217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv" Nov 03 16:05:12.026492 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:12.026467 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42dc28cf-a10c-43ae-b868-4d4a79bab217-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fhfnv\" (UID: \"42dc28cf-a10c-43ae-b868-4d4a79bab217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv" Nov 03 16:05:12.033747 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:12.033721 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42dc28cf-a10c-43ae-b868-4d4a79bab217-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fhfnv\" (UID: \"42dc28cf-a10c-43ae-b868-4d4a79bab217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv" Nov 03 16:05:12.135078 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:12.135035 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjfw\" (UniqueName: \"kubernetes.io/projected/42dc28cf-a10c-43ae-b868-4d4a79bab217-kube-api-access-pmjfw\") pod \"maas-api-revoke-sub-3e31a506-fhfnv\" (UID: \"42dc28cf-a10c-43ae-b868-4d4a79bab217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv" Nov 03 16:05:12.139697 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:12.139658 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42dc28cf-a10c-43ae-b868-4d4a79bab217-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fhfnv\" (UID: \"42dc28cf-a10c-43ae-b868-4d4a79bab217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv" Nov 03 16:05:12.139962 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:12.139943 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42dc28cf-a10c-43ae-b868-4d4a79bab217-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fhfnv\" (UID: \"42dc28cf-a10c-43ae-b868-4d4a79bab217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv" Nov 03 16:05:12.141415 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:12.141392 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42dc28cf-a10c-43ae-b868-4d4a79bab217-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fhfnv\" (UID: \"42dc28cf-a10c-43ae-b868-4d4a79bab217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv" Nov 03 16:05:12.154635 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:12.152056 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42dc28cf-a10c-43ae-b868-4d4a79bab217-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fhfnv\" (UID: \"42dc28cf-a10c-43ae-b868-4d4a79bab217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv" Nov 03 16:05:12.215604 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:12.213432 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjfw\" (UniqueName: \"kubernetes.io/projected/42dc28cf-a10c-43ae-b868-4d4a79bab217-kube-api-access-pmjfw\") pod \"maas-api-revoke-sub-3e31a506-fhfnv\" (UID: \"42dc28cf-a10c-43ae-b868-4d4a79bab217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv" Nov 03 16:05:12.257098 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:12.257045 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv" Nov 03 16:05:12.701625 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:12.695281 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv"] Nov 03 16:05:12.727400 ip-10-0-139-207 kubenswrapper[4557]: W1103 16:05:12.727341 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42dc28cf_a10c_43ae_b868_4d4a79bab217.slice/crio-ff7e7cb06bfb3dae1e7e7808bfb97404db98374b04e476b79e6bcf905eeba5d5 WatchSource:0}: Error finding container ff7e7cb06bfb3dae1e7e7808bfb97404db98374b04e476b79e6bcf905eeba5d5: Status 404 returned error can't find the container with id ff7e7cb06bfb3dae1e7e7808bfb97404db98374b04e476b79e6bcf905eeba5d5 Nov 03 16:05:13.164828 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:13.164790 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv" event={"ID":"42dc28cf-a10c-43ae-b868-4d4a79bab217","Type":"ContainerStarted","Data":"ad0d84010d7024b6d8ab2d3dfbe8ea907127b4d3079e32bca28e5f5594f41f3b"} Nov 03 16:05:13.165280 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:13.164855 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv" event={"ID":"42dc28cf-a10c-43ae-b868-4d4a79bab217","Type":"ContainerStarted","Data":"ff7e7cb06bfb3dae1e7e7808bfb97404db98374b04e476b79e6bcf905eeba5d5"} Nov 03 16:05:13.294256 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:13.294201 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv" podStartSLOduration=2.294183457 podStartE2EDuration="2.294183457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:05:13.292784808 +0000 UTC m=+3423.135906351" watchObservedRunningTime="2026-11-03 16:05:13.294183457 +0000 UTC m=+3423.137305000" Nov 03 16:05:13.805090 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:05:13.805066 4557 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42dc28cf_a10c_43ae_b868_4d4a79bab217.slice/crio-ad0d84010d7024b6d8ab2d3dfbe8ea907127b4d3079e32bca28e5f5594f41f3b.scope\": RecentStats: unable to find data in memory cache]" Nov 03 16:05:14.170597 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:14.170549 4557 generic.go:356] "Generic (PLEG): container finished" podID="42dc28cf-a10c-43ae-b868-4d4a79bab217" containerID="ad0d84010d7024b6d8ab2d3dfbe8ea907127b4d3079e32bca28e5f5594f41f3b" exitCode=0 Nov 03 16:05:14.170962 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:14.170623 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv" event={"ID":"42dc28cf-a10c-43ae-b868-4d4a79bab217","Type":"ContainerDied","Data":"ad0d84010d7024b6d8ab2d3dfbe8ea907127b4d3079e32bca28e5f5594f41f3b"} Nov 03 16:05:15.310897 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:15.310868 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv" Nov 03 16:05:15.389025 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:15.388996 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42dc28cf-a10c-43ae-b868-4d4a79bab217-kube-api-access-pmjfw\" (UniqueName: \"kubernetes.io/projected/42dc28cf-a10c-43ae-b868-4d4a79bab217-kube-api-access-pmjfw\") pod \"42dc28cf-a10c-43ae-b868-4d4a79bab217\" (UID: \"42dc28cf-a10c-43ae-b868-4d4a79bab217\") " Nov 03 16:05:15.389194 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:15.389037 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42dc28cf-a10c-43ae-b868-4d4a79bab217-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42dc28cf-a10c-43ae-b868-4d4a79bab217-maas-api-xks-ca\") pod \"42dc28cf-a10c-43ae-b868-4d4a79bab217\" (UID: \"42dc28cf-a10c-43ae-b868-4d4a79bab217\") " Nov 03 16:05:15.389247 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:15.389209 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42dc28cf-a10c-43ae-b868-4d4a79bab217-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42dc28cf-a10c-43ae-b868-4d4a79bab217-maas-api-openshift-ca\") pod \"42dc28cf-a10c-43ae-b868-4d4a79bab217\" (UID: \"42dc28cf-a10c-43ae-b868-4d4a79bab217\") " Nov 03 16:05:15.389562 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:15.389541 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42dc28cf-a10c-43ae-b868-4d4a79bab217-maas-api-openshift-ca" pod "42dc28cf-a10c-43ae-b868-4d4a79bab217" (UID: "42dc28cf-a10c-43ae-b868-4d4a79bab217"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:05:15.391097 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:15.391075 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42dc28cf-a10c-43ae-b868-4d4a79bab217-maas-api-xks-ca" pod "42dc28cf-a10c-43ae-b868-4d4a79bab217" (UID: "42dc28cf-a10c-43ae-b868-4d4a79bab217"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:05:15.391097 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:15.391084 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42dc28cf-a10c-43ae-b868-4d4a79bab217-kube-api-access-pmjfw" pod "42dc28cf-a10c-43ae-b868-4d4a79bab217" (UID: "42dc28cf-a10c-43ae-b868-4d4a79bab217"). InnerVolumeSpecName "kube-api-access-pmjfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:05:15.490686 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:15.490613 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42dc28cf-a10c-43ae-b868-4d4a79bab217-maas-api-openshift-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:05:15.490686 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:15.490637 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmjfw\" (UniqueName: \"kubernetes.io/projected/42dc28cf-a10c-43ae-b868-4d4a79bab217-kube-api-access-pmjfw\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:05:15.490686 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:15.490646 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42dc28cf-a10c-43ae-b868-4d4a79bab217-maas-api-xks-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:05:15.592197 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:15.592149 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:05:16.180366 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:16.180331 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv" Nov 03 16:05:16.180612 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:16.180360 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv" event={"ID":"42dc28cf-a10c-43ae-b868-4d4a79bab217","Type":"ContainerDied","Data":"ff7e7cb06bfb3dae1e7e7808bfb97404db98374b04e476b79e6bcf905eeba5d5"} Nov 03 16:05:16.180612 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:16.180417 4557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff7e7cb06bfb3dae1e7e7808bfb97404db98374b04e476b79e6bcf905eeba5d5" Nov 03 16:05:16.529433 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:05:16.529356 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:05:34.354761 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:34.354721 4557 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-qxl5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:05:34.355169 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:34.354800 4557 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-qxl5r" podUID="7a2bc03b-b695-479c-a782-8939e59f1000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:05:46.529680 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:05:46.529644 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:05:57.951093 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:57.951013 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c798699b4-dcllm"] Nov 03 16:05:57.951433 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:57.951403 4557 cpu_manager.go:414] "containerMap: removing container" podUID="42dc28cf-a10c-43ae-b868-4d4a79bab217" containerName="revoke-keys" Nov 03 16:05:57.951433 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:57.951415 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42dc28cf-a10c-43ae-b868-4d4a79bab217" containerName="revoke-keys" Nov 03 16:05:57.951508 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:57.951490 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="42dc28cf-a10c-43ae-b868-4d4a79bab217" containerName="revoke-keys" Nov 03 16:05:57.954777 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:57.954760 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" Nov 03 16:05:58.058341 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.058298 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p84nb\"" Nov 03 16:05:58.058603 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.058370 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 16:05:58.058603 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.058490 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 16:05:58.059977 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.059953 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c798699b4-dcllm"] Nov 03 16:05:58.106881 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.106854 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-metrics-certs\") pod \"maas-api-6c798699b4-dcllm\" (UID: \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" Nov 03 16:05:58.106995 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.106895 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-maas-api-tls\") pod \"maas-api-6c798699b4-dcllm\" (UID: \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" Nov 03 16:05:58.106995 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.106917 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-openshift-service-ca\") pod \"maas-api-6c798699b4-dcllm\" (UID: \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" Nov 03 16:05:58.107080 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.107042 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwppb\" (UniqueName: \"kubernetes.io/projected/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-kube-api-access-fwppb\") pod \"maas-api-6c798699b4-dcllm\" (UID: \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" Nov 03 16:05:58.207672 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.207602 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwppb\" (UniqueName: \"kubernetes.io/projected/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-kube-api-access-fwppb\") pod \"maas-api-6c798699b4-dcllm\" (UID: \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" Nov 03 16:05:58.207792 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.207675 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-metrics-certs\") pod \"maas-api-6c798699b4-dcllm\" (UID: \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" Nov 03 16:05:58.207792 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.207709 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-maas-api-tls\") pod \"maas-api-6c798699b4-dcllm\" (UID: \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" Nov 03 16:05:58.207792 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.207731 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-openshift-service-ca\") pod \"maas-api-6c798699b4-dcllm\" (UID: \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" Nov 03 16:05:58.208474 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.208449 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-openshift-service-ca\") pod \"maas-api-6c798699b4-dcllm\" (UID: \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" Nov 03 16:05:58.210163 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.210137 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-metrics-certs\") pod \"maas-api-6c798699b4-dcllm\" (UID: \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" Nov 03 16:05:58.210253 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.210169 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-maas-api-tls\") pod \"maas-api-6c798699b4-dcllm\" (UID: \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" Nov 03 16:05:58.269777 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.269753 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwppb\" (UniqueName: \"kubernetes.io/projected/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-kube-api-access-fwppb\") pod \"maas-api-6c798699b4-dcllm\" (UID: \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" Nov 03 16:05:58.567209 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.567182 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" Nov 03 16:05:58.937644 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.937622 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c798699b4-dcllm"] Nov 03 16:05:58.940046 ip-10-0-139-207 kubenswrapper[4557]: W1103 16:05:58.940018 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9270f75e_4303_4bf3_9a22_399ea9c4cd0f.slice/crio-fd65913b74bdf676cd719358b610396c628906be7da3e65ab82da65cf3ad7885 WatchSource:0}: Error finding container fd65913b74bdf676cd719358b610396c628906be7da3e65ab82da65cf3ad7885: Status 404 returned error can't find the container with id fd65913b74bdf676cd719358b610396c628906be7da3e65ab82da65cf3ad7885 Nov 03 16:05:58.941337 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:58.941322 4557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:05:59.377318 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:05:59.377284 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" event={"ID":"9270f75e-4303-4bf3-9a22-399ea9c4cd0f","Type":"ContainerStarted","Data":"fd65913b74bdf676cd719358b610396c628906be7da3e65ab82da65cf3ad7885"} Nov 03 16:06:00.382732 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:00.382695 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" event={"ID":"9270f75e-4303-4bf3-9a22-399ea9c4cd0f","Type":"ContainerStarted","Data":"d66ceea8d0b4e6ce3d6d97cd215ce3c7d1137c0414b1a4d192c3894db4e2a040"} Nov 03 16:06:00.383129 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:00.382821 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" Nov 03 16:06:00.502365 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:00.502311 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" podStartSLOduration=2.586239077 podStartE2EDuration="3.502297071s" totalImagesPullingTime="916.057994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:06:00.50206558 +0000 UTC m=+3470.345187125" watchObservedRunningTime="2026-11-03 16:06:00.502297071 +0000 UTC m=+3470.345418611" Nov 03 16:06:04.700638 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:04.700604 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:05.362227 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:05.362191 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:05.554614 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:05.554561 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:06.300112 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.300084 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c798699b4-dcllm"] Nov 03 16:06:06.300536 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.300272 4557 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" podUID="9270f75e-4303-4bf3-9a22-399ea9c4cd0f" containerName="maas-api" containerID="cri-o://d66ceea8d0b4e6ce3d6d97cd215ce3c7d1137c0414b1a4d192c3894db4e2a040" gracePeriod=30 Nov 03 16:06:06.305627 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.305607 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" Nov 03 16:06:06.350065 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.350039 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f5bf8854d-jww44"] Nov 03 16:06:06.353343 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.353326 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" Nov 03 16:06:06.374274 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.374253 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d1c8d62-7f71-434d-9b82-de29ec6a2071-metrics-certs\") pod \"maas-api-f5bf8854d-jww44\" (UID: \"7d1c8d62-7f71-434d-9b82-de29ec6a2071\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" Nov 03 16:06:06.374358 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.374312 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d1c8d62-7f71-434d-9b82-de29ec6a2071-openshift-service-ca\") pod \"maas-api-f5bf8854d-jww44\" (UID: \"7d1c8d62-7f71-434d-9b82-de29ec6a2071\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" Nov 03 16:06:06.374423 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.374410 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtcvp\" (UniqueName: \"kubernetes.io/projected/7d1c8d62-7f71-434d-9b82-de29ec6a2071-kube-api-access-jtcvp\") pod \"maas-api-f5bf8854d-jww44\" (UID: \"7d1c8d62-7f71-434d-9b82-de29ec6a2071\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" Nov 03 16:06:06.374467 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.374433 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d1c8d62-7f71-434d-9b82-de29ec6a2071-maas-api-tls\") pod \"maas-api-f5bf8854d-jww44\" (UID: \"7d1c8d62-7f71-434d-9b82-de29ec6a2071\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" Nov 03 16:06:06.401782 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.401560 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5bf8854d-jww44"] Nov 03 16:06:06.475911 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.475879 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtcvp\" (UniqueName: \"kubernetes.io/projected/7d1c8d62-7f71-434d-9b82-de29ec6a2071-kube-api-access-jtcvp\") pod \"maas-api-f5bf8854d-jww44\" (UID: \"7d1c8d62-7f71-434d-9b82-de29ec6a2071\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" Nov 03 16:06:06.476042 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.475926 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d1c8d62-7f71-434d-9b82-de29ec6a2071-maas-api-tls\") pod \"maas-api-f5bf8854d-jww44\" (UID: \"7d1c8d62-7f71-434d-9b82-de29ec6a2071\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" Nov 03 16:06:06.476042 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.475980 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d1c8d62-7f71-434d-9b82-de29ec6a2071-metrics-certs\") pod \"maas-api-f5bf8854d-jww44\" (UID: \"7d1c8d62-7f71-434d-9b82-de29ec6a2071\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" Nov 03 16:06:06.476169 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.476039 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d1c8d62-7f71-434d-9b82-de29ec6a2071-openshift-service-ca\") pod \"maas-api-f5bf8854d-jww44\" (UID: \"7d1c8d62-7f71-434d-9b82-de29ec6a2071\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" Nov 03 16:06:06.476866 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.476840 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d1c8d62-7f71-434d-9b82-de29ec6a2071-openshift-service-ca\") pod \"maas-api-f5bf8854d-jww44\" (UID: \"7d1c8d62-7f71-434d-9b82-de29ec6a2071\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" Nov 03 16:06:06.478516 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.478490 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d1c8d62-7f71-434d-9b82-de29ec6a2071-maas-api-tls\") pod \"maas-api-f5bf8854d-jww44\" (UID: \"7d1c8d62-7f71-434d-9b82-de29ec6a2071\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" Nov 03 16:06:06.478941 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.478921 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d1c8d62-7f71-434d-9b82-de29ec6a2071-metrics-certs\") pod \"maas-api-f5bf8854d-jww44\" (UID: \"7d1c8d62-7f71-434d-9b82-de29ec6a2071\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" Nov 03 16:06:06.538712 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.538682 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtcvp\" (UniqueName: \"kubernetes.io/projected/7d1c8d62-7f71-434d-9b82-de29ec6a2071-kube-api-access-jtcvp\") pod \"maas-api-f5bf8854d-jww44\" (UID: \"7d1c8d62-7f71-434d-9b82-de29ec6a2071\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" Nov 03 16:06:06.551934 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.551881 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" Nov 03 16:06:06.576715 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.576692 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-metrics-certs\") pod \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\" (UID: \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\") " Nov 03 16:06:06.576838 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.576734 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-maas-api-tls\") pod \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\" (UID: \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\") " Nov 03 16:06:06.576838 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.576767 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-kube-api-access-fwppb\" (UniqueName: \"kubernetes.io/projected/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-kube-api-access-fwppb\") pod \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\" (UID: \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\") " Nov 03 16:06:06.576838 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.576828 4557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-openshift-service-ca\") pod \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\" (UID: \"9270f75e-4303-4bf3-9a22-399ea9c4cd0f\") " Nov 03 16:06:06.577253 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.577232 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-openshift-service-ca" pod "9270f75e-4303-4bf3-9a22-399ea9c4cd0f" (UID: "9270f75e-4303-4bf3-9a22-399ea9c4cd0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:06:06.578734 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.578714 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-metrics-certs" pod "9270f75e-4303-4bf3-9a22-399ea9c4cd0f" (UID: "9270f75e-4303-4bf3-9a22-399ea9c4cd0f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:06:06.579020 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.578992 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-maas-api-tls" pod "9270f75e-4303-4bf3-9a22-399ea9c4cd0f" (UID: "9270f75e-4303-4bf3-9a22-399ea9c4cd0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:06:06.579751 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.579733 4557 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-kube-api-access-fwppb" pod "9270f75e-4303-4bf3-9a22-399ea9c4cd0f" (UID: "9270f75e-4303-4bf3-9a22-399ea9c4cd0f"). InnerVolumeSpecName "kube-api-access-fwppb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:06:06.678183 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.678149 4557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-metrics-certs\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:06:06.678183 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.678176 4557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-maas-api-tls\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:06:06.678183 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.678188 4557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwppb\" (UniqueName: \"kubernetes.io/projected/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-kube-api-access-fwppb\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:06:06.678429 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.678198 4557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9270f75e-4303-4bf3-9a22-399ea9c4cd0f-openshift-service-ca\") on node \"ip-10-0-139-207.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 16:06:06.680060 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:06.680034 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" Nov 03 16:06:07.052994 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:07.052927 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5bf8854d-jww44"] Nov 03 16:06:07.055115 ip-10-0-139-207 kubenswrapper[4557]: W1103 16:06:07.055088 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1c8d62_7f71_434d_9b82_de29ec6a2071.slice/crio-2f7ce739b1bb8ec2230f99e5e32c5eead3c52b23cd4981f0b8af8d74395b2337 WatchSource:0}: Error finding container 2f7ce739b1bb8ec2230f99e5e32c5eead3c52b23cd4981f0b8af8d74395b2337: Status 404 returned error can't find the container with id 2f7ce739b1bb8ec2230f99e5e32c5eead3c52b23cd4981f0b8af8d74395b2337 Nov 03 16:06:07.420095 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:07.420059 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" event={"ID":"7d1c8d62-7f71-434d-9b82-de29ec6a2071","Type":"ContainerStarted","Data":"2f7ce739b1bb8ec2230f99e5e32c5eead3c52b23cd4981f0b8af8d74395b2337"} Nov 03 16:06:07.421419 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:07.421397 4557 generic.go:356] "Generic (PLEG): container finished" podID="9270f75e-4303-4bf3-9a22-399ea9c4cd0f" containerID="d66ceea8d0b4e6ce3d6d97cd215ce3c7d1137c0414b1a4d192c3894db4e2a040" exitCode=0 Nov 03 16:06:07.421528 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:07.421438 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" event={"ID":"9270f75e-4303-4bf3-9a22-399ea9c4cd0f","Type":"ContainerDied","Data":"d66ceea8d0b4e6ce3d6d97cd215ce3c7d1137c0414b1a4d192c3894db4e2a040"} Nov 03 16:06:07.421528 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:07.421446 4557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" Nov 03 16:06:07.421528 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:07.421471 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c798699b4-dcllm" event={"ID":"9270f75e-4303-4bf3-9a22-399ea9c4cd0f","Type":"ContainerDied","Data":"fd65913b74bdf676cd719358b610396c628906be7da3e65ab82da65cf3ad7885"} Nov 03 16:06:07.421528 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:07.421492 4557 scope.go:122] "RemoveContainer" containerID="d66ceea8d0b4e6ce3d6d97cd215ce3c7d1137c0414b1a4d192c3894db4e2a040" Nov 03 16:06:07.429657 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:07.429636 4557 scope.go:122] "RemoveContainer" containerID="d66ceea8d0b4e6ce3d6d97cd215ce3c7d1137c0414b1a4d192c3894db4e2a040" Nov 03 16:06:07.429907 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:06:07.429890 4557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d66ceea8d0b4e6ce3d6d97cd215ce3c7d1137c0414b1a4d192c3894db4e2a040\": container with ID starting with d66ceea8d0b4e6ce3d6d97cd215ce3c7d1137c0414b1a4d192c3894db4e2a040 not found: ID does not exist" containerID="d66ceea8d0b4e6ce3d6d97cd215ce3c7d1137c0414b1a4d192c3894db4e2a040" Nov 03 16:06:07.429963 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:07.429915 4557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66ceea8d0b4e6ce3d6d97cd215ce3c7d1137c0414b1a4d192c3894db4e2a040"} err="failed to get container status \"d66ceea8d0b4e6ce3d6d97cd215ce3c7d1137c0414b1a4d192c3894db4e2a040\": rpc error: code = NotFound desc = could not find container \"d66ceea8d0b4e6ce3d6d97cd215ce3c7d1137c0414b1a4d192c3894db4e2a040\": container with ID starting with d66ceea8d0b4e6ce3d6d97cd215ce3c7d1137c0414b1a4d192c3894db4e2a040 not found: ID does not exist" Nov 03 16:06:07.587820 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:07.587736 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c798699b4-dcllm"] Nov 03 16:06:07.637917 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:07.637888 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c798699b4-dcllm"] Nov 03 16:06:08.284020 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:08.283942 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9270f75e-4303-4bf3-9a22-399ea9c4cd0f" path="/var/lib/kubelet/pods/9270f75e-4303-4bf3-9a22-399ea9c4cd0f/volumes" Nov 03 16:06:08.427917 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:08.427872 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" event={"ID":"7d1c8d62-7f71-434d-9b82-de29ec6a2071","Type":"ContainerStarted","Data":"d3a2a09382033843844ef656927b5e1e08877dbcfaacb7ba8d514419a478e988"} Nov 03 16:06:08.428384 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:08.428044 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" Nov 03 16:06:08.547664 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:08.547603 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" podStartSLOduration=1.6460696339999998 podStartE2EDuration="2.547567803s" totalImagesPullingTime="901.498169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:06:08.546528359 +0000 UTC m=+3478.389649901" watchObservedRunningTime="2026-11-03 16:06:08.547567803 +0000 UTC m=+3478.390689345" Nov 03 16:06:08.569510 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:08.569486 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:08.879950 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:08.879875 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:11.863372 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:11.863333 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:12.059755 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:12.059720 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:13.447528 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:13.447489 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:13.641403 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:13.641372 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:13.839267 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:13.839236 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:14.436965 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:14.436934 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f5bf8854d-jww44" Nov 03 16:06:15.960774 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:15.960715 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:16.529224 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:06:16.529186 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:06:17.326245 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:17.326215 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:18.061763 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:18.061728 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:18.853357 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:18.853325 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:23.725308 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:23.725164 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:28.580550 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:28.580511 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:33.803530 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:33.803487 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:39.040990 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:39.040946 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:43.919436 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:43.919381 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:46.530709 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:06:46.530673 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:06:46.713290 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:46.713254 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:46.899541 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:46.899502 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:48.704567 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:48.704509 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:53.022242 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:53.022205 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:54.572761 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:54.572724 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:54.849837 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:54.849758 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:56.180877 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:56.180841 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:57.519161 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:57.519124 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:58.292665 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:58.292610 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:06:58.659951 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:06:58.659911 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:02.496860 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:02.496813 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:02.712731 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:02.712681 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:02.988650 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:02.988607 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:05.792262 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:05.792221 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:05.981462 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:05.981429 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:09.859102 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:09.859066 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:10.059717 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:10.059682 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:10.261664 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:10.261573 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:14.925200 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:14.925135 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:16.311493 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:16.311457 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:16.529180 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:07:16.529138 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:07:20.020143 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:20.020111 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:24.910620 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:24.910512 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:27.517686 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:27.517649 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:30.145453 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:30.145420 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:35.008594 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:35.008562 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:35.902894 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:35.902860 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:40.113742 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:40.113704 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:44.998321 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:44.998284 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:46.528782 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:07:46.528747 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:07:49.912753 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:49.912712 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:54.812903 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:54.812864 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:07:59.723491 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:07:59.723440 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:08:04.621530 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:08:04.621443 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:08:09.524117 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:08:09.524073 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:08:10.701461 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:08:10.701435 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 16:08:10.709197 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:08:10.709172 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 16:08:14.428344 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:08:14.428310 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:08:16.528898 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:08:16.528861 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:08:16.665646 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:08:16.665615 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:08:19.326861 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:08:19.326821 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:08:24.230568 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:08:24.230531 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:08:29.187336 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:08:29.187305 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:08:34.113029 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:08:34.112990 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:08:39.033282 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:08:39.033243 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:08:43.917691 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:08:43.917655 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:08:46.528538 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:08:46.528501 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:08:48.926524 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:08:48.926480 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:08:54.451617 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:08:54.451521 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:08:59.431922 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:08:59.431884 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:09:00.433487 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:09:00.433446 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:09:00.625893 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:09:00.625856 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:09:04.226976 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:09:04.226947 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:09:13.579945 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:09:13.579893 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:09:15.089577 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:09:15.089547 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:09:15.364855 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:09:15.364742 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:09:16.177041 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:09:16.176994 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:09:16.529221 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:09:16.529130 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:09:17.013199 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:09:17.013166 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:09:17.377428 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:09:17.377389 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:09:22.558768 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:09:22.558724 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:09:22.828061 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:09:22.827979 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:09:23.042517 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:09:23.042485 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:09:46.527987 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:09:46.527951 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:09:47.575423 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:09:47.575326 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:09:47.759301 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:09:47.759267 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:16.049006 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:10:16.048973 4557 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv"] Nov 03 16:10:16.099092 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:10:16.099059 4557 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhfnv"] Nov 03 16:10:16.284021 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:10:16.283987 4557 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42dc28cf-a10c-43ae-b868-4d4a79bab217" path="/var/lib/kubelet/pods/42dc28cf-a10c-43ae-b868-4d4a79bab217/volumes" Nov 03 16:10:16.529439 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:10:16.529363 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:10:17.372418 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:10:17.372382 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:37.573602 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:10:37.573463 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:37.783511 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:10:37.783475 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:45.055573 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:10:45.055539 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:10:46.529199 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:10:46.529162 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:10:48.033315 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:10:48.033281 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6n4n9_006c2584-8b0a-48d1-ad51-14839b7ab01f/manager/0.log" Nov 03 16:10:48.632244 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:10:48.632221 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xhpgx_9a00bc65-dafa-429d-ba69-4dd8b9c8cbac/manager/0.log" Nov 03 16:10:50.116148 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:10:50.116117 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2k2m2_9e6b3b69-64db-4006-8f4a-e78bd1e8cc05/manager/0.log" Nov 03 16:10:51.450391 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:10:51.450361 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f5bf8854d-jww44_7d1c8d62-7f71-434d-9b82-de29ec6a2071/maas-api/0.log" Nov 03 16:10:52.892801 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:10:52.892772 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-8f78l_e0e3856c-467a-4e0a-bec1-0873dc8bb5c2/postgres/0.log" Nov 03 16:10:55.316259 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:10:55.316228 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-bcvlh_7ea8b6c2-0d66-41bc-b592-6acb736f0246/manager/0.log" Nov 03 16:10:55.586075 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:10:55.586002 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-s5mfs_fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0/kuadrant-console-plugin/0.log" Nov 03 16:10:56.624251 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:10:56.624222 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4c2w8_6b055afd-c279-404c-8554-4a7444bfb519/manager/0.log" Nov 03 16:10:58.442500 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:10:58.442466 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68fb7df4dc-28mfp_809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4/kube-auth-proxy/0.log" Nov 03 16:10:59.266379 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:10:59.266348 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-5952j_6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e/istio-proxy/0.log" Nov 03 16:11:00.212930 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:11:00.212901 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-9x97t_8345cb5f-d59d-44a3-b858-5885db9bb03f/istio-proxy/0.log" Nov 03 16:11:01.224309 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:11:01.224279 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-qxl5r_7a2bc03b-b695-479c-a782-8939e59f1000/istio-proxy/0.log" Nov 03 16:11:16.529678 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:11:16.529638 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:11:17.371064 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:11:17.371027 4557 scope.go:122] "RemoveContainer" containerID="ad0d84010d7024b6d8ab2d3dfbe8ea907127b4d3079e32bca28e5f5594f41f3b" Nov 03 16:11:17.622941 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:11:17.622864 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6n4n9_006c2584-8b0a-48d1-ad51-14839b7ab01f/manager/0.log" Nov 03 16:11:17.726207 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:11:17.726161 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:11:18.223470 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:11:18.223429 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xhpgx_9a00bc65-dafa-429d-ba69-4dd8b9c8cbac/manager/0.log" Nov 03 16:11:19.259395 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:11:19.259361 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2k2m2_9e6b3b69-64db-4006-8f4a-e78bd1e8cc05/manager/0.log" Nov 03 16:11:20.292508 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:11:20.292446 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f5bf8854d-jww44_7d1c8d62-7f71-434d-9b82-de29ec6a2071/maas-api/0.log" Nov 03 16:11:21.638271 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:11:21.638226 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-8f78l_e0e3856c-467a-4e0a-bec1-0873dc8bb5c2/postgres/0.log" Nov 03 16:11:23.644674 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:11:23.644606 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-bcvlh_7ea8b6c2-0d66-41bc-b592-6acb736f0246/manager/0.log" Nov 03 16:11:23.929056 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:11:23.928980 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-s5mfs_fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0/kuadrant-console-plugin/0.log" Nov 03 16:11:24.736855 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:11:24.736824 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4c2w8_6b055afd-c279-404c-8554-4a7444bfb519/manager/0.log" Nov 03 16:11:26.153150 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:11:26.153122 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68fb7df4dc-28mfp_809d8f5b-51ed-4b00-a9fb-bd57ab23d5b4/kube-auth-proxy/0.log" Nov 03 16:11:26.965299 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:11:26.965261 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-5952j_6d7cfe96-7fdb-4ab2-b140-89d8acc9ab2e/istio-proxy/0.log" Nov 03 16:11:27.776502 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:11:27.776476 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-9x97t_8345cb5f-d59d-44a3-b858-5885db9bb03f/istio-proxy/0.log" Nov 03 16:11:28.739453 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:11:28.739425 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-qxl5r_7a2bc03b-b695-479c-a782-8939e59f1000/istio-proxy/0.log" Nov 03 16:11:46.529748 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:11:46.529707 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:11:48.846353 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:11:48.846320 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:16.528369 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:12:16.528333 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:12:18.096537 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:12:18.096498 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:18.525280 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:12:18.525192 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:46.528774 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:12:46.528732 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:12:55.470666 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:12:55.470627 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:12:55.656040 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:12:55.656013 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:13:04.438139 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:04.438099 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7vw5n_a1360ffe-4b52-4b77-8fbc-7a74c3b052a3/global-pull-secret-syncer/0.log" Nov 03 16:13:07.768652 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:07.768623 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" Nov 03 16:13:09.466759 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:09.466695 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-207.us-west-1.compute.internal_abb209747bf3463921ac94593d5ab068/haproxy/0.log" Nov 03 16:13:10.730572 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:10.730543 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 16:13:10.740098 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:10.740077 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmvs8_731f41d1-b212-4b32-b0e1-b4d4188ed782/ovn-acl-logging/0.log" Nov 03 16:13:14.648458 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.648416 4557 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7"] Nov 03 16:13:14.648879 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.648809 4557 cpu_manager.go:414] "containerMap: removing container" podUID="9270f75e-4303-4bf3-9a22-399ea9c4cd0f" containerName="maas-api" Nov 03 16:13:14.648879 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.648822 4557 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9270f75e-4303-4bf3-9a22-399ea9c4cd0f" containerName="maas-api" Nov 03 16:13:14.648962 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.648905 4557 memory_manager.go:365] "RemoveStaleState removing state" podUID="9270f75e-4303-4bf3-9a22-399ea9c4cd0f" containerName="maas-api" Nov 03 16:13:14.652030 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.652014 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:14.724673 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.724633 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrgqn\" (UniqueName: \"kubernetes.io/projected/35cfd896-48bd-4cf5-8f27-7f8514223990-kube-api-access-hrgqn\") pod \"perf-node-gather-daemonset-68tq7\" (UID: \"35cfd896-48bd-4cf5-8f27-7f8514223990\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:14.724837 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.724693 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/35cfd896-48bd-4cf5-8f27-7f8514223990-podres\") pod \"perf-node-gather-daemonset-68tq7\" (UID: \"35cfd896-48bd-4cf5-8f27-7f8514223990\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:14.724837 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.724773 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35cfd896-48bd-4cf5-8f27-7f8514223990-lib-modules\") pod \"perf-node-gather-daemonset-68tq7\" (UID: \"35cfd896-48bd-4cf5-8f27-7f8514223990\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:14.724837 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.724797 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35cfd896-48bd-4cf5-8f27-7f8514223990-sys\") pod \"perf-node-gather-daemonset-68tq7\" (UID: \"35cfd896-48bd-4cf5-8f27-7f8514223990\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:14.724949 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.724890 4557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/35cfd896-48bd-4cf5-8f27-7f8514223990-proc\") pod \"perf-node-gather-daemonset-68tq7\" (UID: \"35cfd896-48bd-4cf5-8f27-7f8514223990\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:14.745255 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.745219 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7"] Nov 03 16:13:14.792771 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.792740 4557 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-76mwr\"/\"default-dockercfg-msnrf\"" Nov 03 16:13:14.792771 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.792745 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-76mwr\"/\"openshift-service-ca.crt\"" Nov 03 16:13:14.809225 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.809201 4557 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-76mwr\"/\"kube-root-ca.crt\"" Nov 03 16:13:14.825362 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.825335 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrgqn\" (UniqueName: \"kubernetes.io/projected/35cfd896-48bd-4cf5-8f27-7f8514223990-kube-api-access-hrgqn\") pod \"perf-node-gather-daemonset-68tq7\" (UID: \"35cfd896-48bd-4cf5-8f27-7f8514223990\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:14.825489 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.825380 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/35cfd896-48bd-4cf5-8f27-7f8514223990-podres\") pod \"perf-node-gather-daemonset-68tq7\" (UID: \"35cfd896-48bd-4cf5-8f27-7f8514223990\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:14.825489 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.825404 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35cfd896-48bd-4cf5-8f27-7f8514223990-lib-modules\") pod \"perf-node-gather-daemonset-68tq7\" (UID: \"35cfd896-48bd-4cf5-8f27-7f8514223990\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:14.825489 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.825424 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35cfd896-48bd-4cf5-8f27-7f8514223990-sys\") pod \"perf-node-gather-daemonset-68tq7\" (UID: \"35cfd896-48bd-4cf5-8f27-7f8514223990\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:14.825489 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.825461 4557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/35cfd896-48bd-4cf5-8f27-7f8514223990-proc\") pod \"perf-node-gather-daemonset-68tq7\" (UID: \"35cfd896-48bd-4cf5-8f27-7f8514223990\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:14.825704 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.825479 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/35cfd896-48bd-4cf5-8f27-7f8514223990-podres\") pod \"perf-node-gather-daemonset-68tq7\" (UID: \"35cfd896-48bd-4cf5-8f27-7f8514223990\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:14.825704 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.825547 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/35cfd896-48bd-4cf5-8f27-7f8514223990-proc\") pod \"perf-node-gather-daemonset-68tq7\" (UID: \"35cfd896-48bd-4cf5-8f27-7f8514223990\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:14.825704 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.825564 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35cfd896-48bd-4cf5-8f27-7f8514223990-lib-modules\") pod \"perf-node-gather-daemonset-68tq7\" (UID: \"35cfd896-48bd-4cf5-8f27-7f8514223990\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:14.825704 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.825547 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35cfd896-48bd-4cf5-8f27-7f8514223990-sys\") pod \"perf-node-gather-daemonset-68tq7\" (UID: \"35cfd896-48bd-4cf5-8f27-7f8514223990\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:14.887086 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.887052 4557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrgqn\" (UniqueName: \"kubernetes.io/projected/35cfd896-48bd-4cf5-8f27-7f8514223990-kube-api-access-hrgqn\") pod \"perf-node-gather-daemonset-68tq7\" (UID: \"35cfd896-48bd-4cf5-8f27-7f8514223990\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:14.963624 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:14.963517 4557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:15.335721 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:15.335662 4557 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7"] Nov 03 16:13:15.337906 ip-10-0-139-207 kubenswrapper[4557]: W1103 16:13:15.337878 4557 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod35cfd896_48bd_4cf5_8f27_7f8514223990.slice/crio-7e353b17e8abb101b50026032a048c356ddae01aa4393d0abed2e8a874d2694a WatchSource:0}: Error finding container 7e353b17e8abb101b50026032a048c356ddae01aa4393d0abed2e8a874d2694a: Status 404 returned error can't find the container with id 7e353b17e8abb101b50026032a048c356ddae01aa4393d0abed2e8a874d2694a Nov 03 16:13:16.023335 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:16.023308 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-bcvlh_7ea8b6c2-0d66-41bc-b592-6acb736f0246/manager/0.log" Nov 03 16:13:16.305981 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:16.305951 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" event={"ID":"35cfd896-48bd-4cf5-8f27-7f8514223990","Type":"ContainerStarted","Data":"1b62f72d84b54ece7a80dfc71b9d3d74327127bf89da6b5bd6dc7b83614bdb56"} Nov 03 16:13:16.306142 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:16.306002 4557 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" event={"ID":"35cfd896-48bd-4cf5-8f27-7f8514223990","Type":"ContainerStarted","Data":"7e353b17e8abb101b50026032a048c356ddae01aa4393d0abed2e8a874d2694a"} Nov 03 16:13:16.306142 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:16.306036 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:16.435693 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:16.435635 4557 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" podStartSLOduration=2.435619556 podStartE2EDuration="2.435619556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:16.434972497 +0000 UTC m=+3906.278094039" watchObservedRunningTime="2026-11-03 16:13:16.435619556 +0000 UTC m=+3906.278741098" Nov 03 16:13:16.479271 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:16.479250 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-s5mfs_fb9b17e7-3939-4d8f-b7c3-29550c8ef8e0/kuadrant-console-plugin/0.log" Nov 03 16:13:16.528452 ip-10-0-139-207 kubenswrapper[4557]: E1103 16:13:16.528416 4557 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x575x" containerName="extractor" Nov 03 16:13:17.840387 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:17.840352 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4c2w8_6b055afd-c279-404c-8554-4a7444bfb519/manager/0.log" Nov 03 16:13:18.462786 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:18.462754 4557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mh4np_c432b1d0-d7d8-49c4-a26f-716b10424bc9/konnectivity-agent/0.log" limit=1024 Nov 03 16:13:22.320024 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:22.319998 4557 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-68tq7" Nov 03 16:13:26.130556 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:26.130523 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2jprl_241a4808-c092-4c87-b9cd-0f88efe30647/node-exporter/0.log" Nov 03 16:13:26.551165 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:26.551138 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2jprl_241a4808-c092-4c87-b9cd-0f88efe30647/kube-rbac-proxy/0.log" Nov 03 16:13:26.975478 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:26.975406 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2jprl_241a4808-c092-4c87-b9cd-0f88efe30647/init-textfile/0.log" Nov 03 16:13:31.418285 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:31.418255 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3c510ec8-7799-4f62-8e3b-7bdd94f95a2e/prometheus/0.log" Nov 03 16:13:31.884320 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:31.884297 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3c510ec8-7799-4f62-8e3b-7bdd94f95a2e/config-reloader/0.log" Nov 03 16:13:32.309935 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:32.309905 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3c510ec8-7799-4f62-8e3b-7bdd94f95a2e/thanos-sidecar/0.log" Nov 03 16:13:32.733923 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:32.733850 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3c510ec8-7799-4f62-8e3b-7bdd94f95a2e/kube-rbac-proxy-web/0.log" Nov 03 16:13:33.154638 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:33.154612 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3c510ec8-7799-4f62-8e3b-7bdd94f95a2e/kube-rbac-proxy/0.log" Nov 03 16:13:33.575303 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:33.575282 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3c510ec8-7799-4f62-8e3b-7bdd94f95a2e/kube-rbac-proxy-thanos/0.log" Nov 03 16:13:33.999182 ip-10-0-139-207 kubenswrapper[4557]: I1103 16:13:33.999118 4557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3c510ec8-7799-4f62-8e3b-7bdd94f95a2e/init-config-reloader/0.log"