Sep 11 21:28:43.748587 ip-10-0-142-211 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 21:28:43.748599 ip-10-0-142-211 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 21:28:43.748607 ip-10-0-142-211 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 21:28:43.748760 ip-10-0-142-211 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 21:28:53.997883 ip-10-0-142-211 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 21:28:53.997900 ip-10-0-142-211 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 264f50fb59b9495086836b25dd4c9cf4 --
Sep 11 21:30:35.611527 ip-10-0-142-211 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 21:30:36.074711 ip-10-0-142-211 kubenswrapper[2544]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:30:36.074711 ip-10-0-142-211 kubenswrapper[2544]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 21:30:36.074711 ip-10-0-142-211 kubenswrapper[2544]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:30:36.074711 ip-10-0-142-211 kubenswrapper[2544]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 21:30:36.074711 ip-10-0-142-211 kubenswrapper[2544]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:30:36.078813 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.078710    2544 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 21:30:36.081961 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.081943    2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:30:36.081961 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.081960    2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:30:36.081961 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.081963    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:30:36.081961 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.081967    2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.081970    2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.081974    2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.081977    2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.081980    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.081983    2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.081987    2544 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.081990    2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.081993    2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.081995    2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.081998    2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082001    2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082004    2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082006    2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082009    2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082012    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082015    2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082017    2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082026    2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082030    2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:30:36.082090 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082032    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082035    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082037    2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082040    2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082043    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082046    2544 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082050    2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082054    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082057    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082060    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082063    2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082066    2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082070    2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082072    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082077    2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082080    2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082083    2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082086    2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082089    2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:30:36.082601 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082092    2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082094    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082097    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082101    2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082104    2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082107    2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082109    2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082112    2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082115    2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082117    2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082120    2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082122    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082127    2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082130    2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082133    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082135    2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082138    2544 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082141    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082143    2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082146    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:30:36.083077 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082149    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082151    2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082154    2544 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082158    2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082162    2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082167    2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082169    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082173    2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082175    2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082178    2544 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082181    2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082184    2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082186    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082201    2544 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082206    2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082209    2544 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082212    2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082216    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082218    2544 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082222    2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:30:36.083604 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082225    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082228    2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082231    2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082233    2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082237    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082240    2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082243    2544 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082246    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082249    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082251    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082254    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082256    2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082259    2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082262    2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082265    2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082267    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082270    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082977    2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082984    2544 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082987    2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:30:36.084149 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082990    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082993    2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.082998    2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083004    2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083007    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083010    2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083013    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083016    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083019    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083022    2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083024    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083027    2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083029    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083032    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083035    2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083038    2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083040    2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083043    2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083046    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:30:36.084656 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083049    2544 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083051    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083054    2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083057    2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083059    2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083063    2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083066    2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083069    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083072    2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083074    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083077    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083079    2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083083    2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083085    2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083088    2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083091    2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083093    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083096    2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083099    2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:30:36.085363 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083101    2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083104    2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083106    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083109    2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083112    2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083114    2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083116    2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083119    2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083121    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083125    2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083128    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083131    2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083134    2544 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083137    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083141    2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083143    2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083146    2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083148    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083153    2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:30:36.086029 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083156    2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083159    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083161    2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083164    2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083167    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083169    2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083172    2544 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083174    2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083177    2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083180    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083182    2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083185    2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083187    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083205    2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083207    2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083210    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083213    2544 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083216    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083220    2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:30:36.086535 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083223    2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083226    2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083229    2544 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083235    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083238    2544 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083241    2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083244    2544 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083253    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083257    2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083260    2544 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083264    2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083267    2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083270    2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083273    2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083275    2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083278    2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083280    2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083283    2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083286    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083288    2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:30:36.087015 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083676    2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083680    2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083683    2544 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083686    2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083689    2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083692    2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083695    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083698    2544 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083701    2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083704    2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083706    2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083709    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083712    2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083714    2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083717    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083720    2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083723    2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083726    2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083729    2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:30:36.087539 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083731    2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083736    2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083740    2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083743    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083746    2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083749    2544 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083752    2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083754    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083757    2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083760    2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083762    2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083764    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083767    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083769    2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083772    2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083775    2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083778    2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083782    2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083784    2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:30:36.088005 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083787    2544 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083789    2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083792    2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083795    2544 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083797    2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083799    2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083802    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083804    2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083807    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083810    2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083813    2544 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083816    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083819    2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083821    2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083824    2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083827    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083829    2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083833    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083836    2544 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083838    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:30:36.088505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083840    2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083843    2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083845    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083848    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083851    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083853    2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083856    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083858    2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083861    2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083863    2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083866    2544 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083868    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083870    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083873    2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083875    2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083878    2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083880    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083883    2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083885    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083888    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:30:36.089155 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083892    2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083896    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083900    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083902    2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083905    2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083908    2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083911    2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083914    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083917    2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083919    2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083922    2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083925    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083927    2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083929    2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083932    2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083935    2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083937    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083939    2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083942    2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083944    2544 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:30:36.089740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.083947    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084013    2544 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084052    2544 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084059    2544 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084064    2544 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084068    2544 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084072    2544 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084082    2544 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084088    2544 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084091    2544 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084095    2544 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084098    2544 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084102    2544 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084106    2544 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084109    2544 flags.go:64] FLAG: --cgroup-root=""
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084112    2544 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084115    2544 flags.go:64] FLAG: --client-ca-file=""
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084118    2544 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084121    2544 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084127    2544 flags.go:64] FLAG: --cluster-domain=""
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084130    2544 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084134    2544 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084137    2544 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084140    2544 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 21:30:36.090394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084144    2544 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084148    2544 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084151    2544 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084155    2544 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084158    2544 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084161    2544 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084165    2544 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084168    2544 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084171    2544 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084175    2544 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084178    2544 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084182    2544 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084185    2544 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084188    2544 flags.go:64] FLAG: --enable-server="true"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084212    2544 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084217    2544 flags.go:64] FLAG: --event-burst="100"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084220    2544 flags.go:64] FLAG: --event-qps="50"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084223    2544 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084226    2544 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084229    2544 flags.go:64] FLAG: --eviction-hard=""
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084233    2544 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084238    2544 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084241    2544 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084244    2544 flags.go:64] FLAG: --eviction-soft=""
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084248    2544 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 21:30:36.090984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084251    2544 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084254    2544 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084257    2544 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084260    2544 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084263    2544 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084267    2544 flags.go:64] FLAG: --feature-gates=""
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084271    2544 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084274    2544 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084277    2544 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084280    2544 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084283    2544 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084286    2544 flags.go:64] FLAG: --help="false"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084289    2544 flags.go:64] FLAG: --hostname-override="ip-10-0-142-211.ec2.internal"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084292    2544 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084295    2544 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084299    2544 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084302    2544 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084305    2544 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084308    2544 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084311    2544 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084315    2544 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084317    2544 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084321    2544 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084325    2544 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 21:30:36.091647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084328    2544 flags.go:64] FLAG: --kube-reserved=""
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084331    2544 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084334    2544 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084337    2544 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084342    2544 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084345    2544 flags.go:64] FLAG: --lock-file=""
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084348    2544 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084351    2544 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084354    2544 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084360    2544 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084363    2544 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084366    2544 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084370    2544 flags.go:64] FLAG: --logging-format="text"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084373    2544 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084377    2544 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084380    2544 flags.go:64] FLAG: --manifest-url=""
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084383    2544 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084388    2544 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084391    2544 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084400    2544 flags.go:64] FLAG: --max-pods="110"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084403    2544 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084406    2544 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084409    2544 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084412    2544 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084415    2544 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 21:30:36.092283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084418    2544 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084422    2544 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084429    2544 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084433    2544 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084436    2544 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084441    2544 flags.go:64] FLAG: --pod-cidr=""
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084444    2544 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084450    2544 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084454    2544 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084457    2544 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084460    2544 flags.go:64] FLAG: --port="10250"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084464    2544 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084467    2544 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-010cded6c3bf27dab"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084471    2544 flags.go:64] FLAG: --qos-reserved=""
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084474    2544 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084477    2544 flags.go:64] FLAG: --register-node="true"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084480    2544 flags.go:64] FLAG: --register-with-taints=""
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084484    2544 flags.go:64] FLAG: --registry-burst="10"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084487    2544 flags.go:64] FLAG: --registry-qps="5"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084490    2544 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084493    2544 flags.go:64] FLAG: --reserved-memory=""
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084497    2544 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084501    2544 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084504    2544 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084507    2544 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 21:30:36.092924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084510    2544 flags.go:64] FLAG: --runonce="false"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084513    2544 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084516    2544 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084519    2544 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084523    2544 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084526    2544 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084529    2544 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084532    2544 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084535    2544 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084538    2544 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084542    2544 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084544    2544 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084547    2544 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084553    2544 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084556    2544 flags.go:64] FLAG: --system-cgroups=""
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084559    2544 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084565    2544 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084568    2544 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084573    2544 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084577    2544 flags.go:64] FLAG: --tls-min-version=""
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084580    2544 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084583    2544 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084586    2544 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084589    2544 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084592    2544 flags.go:64] FLAG: --v="2"
Sep 11 21:30:36.093647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084597    2544 flags.go:64] FLAG: --version="false"
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084601    2544 flags.go:64] FLAG: --vmodule=""
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084605    2544 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.084609    2544 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.084726    2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.084731    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.084734    2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.084737    2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.084740    2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.084742    2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.084745    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.084747    2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.084750    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.084753    2544 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.084756    2544 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.084758    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085882    2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085895    2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085899    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085903    2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085909    2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:30:36.094341 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085912    2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085916    2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085919    2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085922    2544 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085927    2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085930    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085935    2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085943    2544 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085946    2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085949    2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085953    2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085957    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085960    2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085963    2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085966    2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085969    2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085972    2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085987    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085990    2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:30:36.094874 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.085994    2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086000    2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086002    2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086005    2544 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086008    2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086011    2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086014    2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086017    2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086020    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086024    2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086026    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086030    2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086034    2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086037    2544 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086043    2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086046    2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086049    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086053    2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086056    2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086059    2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:30:36.095380 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086062    2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086065    2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086067    2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086071    2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086074    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086076    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086082    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086085    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086088    2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086091    2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086094    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086098    2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086103    2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086106    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086109    2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086112    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086116    2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086120    2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086125    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:30:36.095886 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086128    2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086132    2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086135    2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086138    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086142    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086147    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086149    2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086152    2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086156    2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086158    2544 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086163    2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086170    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086173    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086176    2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086179    2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086182    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086185    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086188    2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086206    2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086209    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:30:36.096387 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086212    2544 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:30:36.096925 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086215    2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:30:36.096925 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086218    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:30:36.096925 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.086221    2544 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:30:36.096925 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.087048    2544 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:30:36.096925 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.094693    2544 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 21:30:36.096925 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.094712    2544 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 21:30:36.096925 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.094733    2544 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 21:30:36.096925 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.094739    2544 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 21:30:36.096925 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094796    2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:30:36.096925 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094801    2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:30:36.096925 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094805    2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:30:36.096925 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094808    2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:30:36.096925 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094811    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:30:36.096925 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094814    2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094817    2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094820    2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094822    2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094825    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094828    2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094831    2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094833    2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094836    2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094839    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094843    2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094845    2544 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094848    2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094851    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094853    2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094856    2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094859    2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094861    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094864    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094868    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:30:36.097303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094871    2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094873    2544 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094876    2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094879    2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094883    2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094886    2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094888    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094891    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094894    2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094897    2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094900    2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094902    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094905    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094907    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094910    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094913    2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094915    2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094918    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094921    2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094923    2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:30:36.097815 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094926    2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094929    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094932    2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094935    2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094937    2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094940    2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094943    2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094945    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094948    2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094950    2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094953    2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094956    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094959    2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094961    2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094964    2544 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094968    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094971    2544 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094976    2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094980    2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:30:36.098377 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094983    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094987    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094990    2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094993    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094996    2544 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.094998    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095002    2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095006    2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095009    2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095012    2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095015    2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095017    2544 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095020    2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095023    2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095027    2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095030    2544 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095033    2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095036    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095039    2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095041    2544 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:30:36.098856 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095044    2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:30:36.099390 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095047    2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:30:36.099390 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095049    2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:30:36.099390 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095052    2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:30:36.099390 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095054    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:30:36.099390 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095058    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:30:36.099390 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095061    2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:30:36.099390 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095063    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:30:36.099390 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095067    2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:30:36.099390 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095070    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:30:36.099390 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095073    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:30:36.099390 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095077    2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:30:36.099390 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095080    2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:30:36.099390 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095083    2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:30:36.099390 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095085    2544 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:30:36.099390 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.095091    2544 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095243    2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095257    2544 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095261    2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095267    2544 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095270    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095273    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095277    2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095279    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095283    2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095286    2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095289    2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095292    2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095294    2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095298    2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095301    2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095304    2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095306    2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095309    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:30:36.099763 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095312    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095315    2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095317    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095320    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095323    2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095326    2544 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095330    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095333    2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095336    2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095339    2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095341    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095344    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095347    2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095349    2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095352    2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095355    2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095359    2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095362    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095365    2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:30:36.100231 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095368    2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095371    2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095374    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095376    2544 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095379    2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095381    2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095384    2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095387    2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095390    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095393    2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095396    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095398    2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095401    2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095404    2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095406    2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095409    2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095411    2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095414    2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095419    2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095422    2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:30:36.100710 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095425    2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095428    2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095431    2544 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095434    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095437    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095440    2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095443    2544 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095446    2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095448    2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095451    2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095454    2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095456    2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095459    2544 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095462    2544 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095465    2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095468    2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095470    2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095473    2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095476    2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095478    2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:30:36.101212 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095481    2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095484    2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095487    2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095489    2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095492    2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095495    2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095497    2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095500    2544 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095503    2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095506    2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095508    2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095512    2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095515    2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095518    2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095520    2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095524    2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095526    2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095530    2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095534    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095537    2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:30:36.101716 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095540    2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:30:36.102340 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:36.095543    2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:30:36.102340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.095547    2544 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:30:36.102340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.095683    2544 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 21:30:36.102340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.097794    2544 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 21:30:36.102340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.098812    2544 server.go:1019] "Starting client certificate rotation"
Sep 11 21:30:36.102340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.098910    2544 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 21:30:36.102340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.098947    2544 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 21:30:36.102340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.101954    2544 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 21:30:36.103177 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.103156    2544 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 21:30:36.114348 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.114328    2544 log.go:25] "Validated CRI v1 runtime API"
Sep 11 21:30:36.118502 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.118481    2544 log.go:25] "Validated CRI v1 image API"
Sep 11 21:30:36.119374 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.119352    2544 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 21:30:36.121902 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.121881    2544 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7ac7182a-579b-4c04-8545-c1a2cd468ead:/dev/nvme0n1p3 f41a9208-6f8d-4f89-9bc0-796aa920e96f:/dev/nvme0n1p4]
Sep 11 21:30:36.121980 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.121902    2544 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 21:30:36.129254 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.129116    2544 manager.go:216] Machine: {Timestamp:2026-09-11 21:30:36.12763844 +0000 UTC m=+0.375081957 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103034 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29647f3b3a9191bb5da7525a8cb903 SystemUUID:ec29647f-3b3a-9191-bb5d-a7525a8cb903 BootID:264f50fb-59b9-4950-8683-6b25dd4c9cf4 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:25:9d:7c:b2:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:25:9d:7c:b2:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:7f:91:7d:c5:f8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 21:30:36.129254 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.129236    2544 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 21:30:36.129408 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.129326    2544 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 21:30:36.133331 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.133292    2544 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 21:30:36.133475 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.133333    2544 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-211.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 21:30:36.135200 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.135181    2544 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 21:30:36.135255 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.135204    2544 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 21:30:36.135255 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.135213    2544 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 21:30:36.135810 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.135799    2544 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 21:30:36.136401 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.136380    2544 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 21:30:36.136714 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.136699    2544 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 21:30:36.141138 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.141126    2544 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 21:30:36.141280 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.141270    2544 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 21:30:36.141333 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.141301    2544 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 21:30:36.141333 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.141311    2544 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 21:30:36.141333 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.141326    2544 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 21:30:36.141455 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.141336    2544 kubelet.go:398] "Adding apiserver pod source"
Sep 11 21:30:36.141455 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.141345    2544 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 21:30:36.141733 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.141720    2544 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 21:30:36.141786 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.141737    2544 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 21:30:36.144652 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.144636    2544 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 21:30:36.147110 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.147094    2544 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 21:30:36.147159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.147135    2544 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 21:30:36.149676 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.149661    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 21:30:36.149718 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.149686    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 21:30:36.149718 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.149695    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 21:30:36.149718 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.149704    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 21:30:36.149718 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.149719    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 21:30:36.149824 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.149727    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 21:30:36.149824 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.149746    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 21:30:36.149824 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.149755    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 21:30:36.149824 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.149764    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 21:30:36.149824 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.149771    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 21:30:36.149824 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.149785    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 21:30:36.149824 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.149794    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 21:30:36.150724 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.150713    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 21:30:36.150770 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.150726    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 21:30:36.152105 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:36.152082    2544 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-211.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 21:30:36.152177 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:36.152082    2544 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 21:30:36.152322 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.152305    2544 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-211.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 21:30:36.155376 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.155358    2544 server.go:1268] "Started kubelet"
Sep 11 21:30:36.155486 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.155454    2544 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 21:30:36.155723 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.155569    2544 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 21:30:36.155845 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.155831    2544 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 21:30:36.156386 ip-10-0-142-211 systemd[1]: Started Kubernetes Kubelet.
Sep 11 21:30:36.158287 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.158271    2544 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 21:30:36.160217 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.160154    2544 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 21:30:36.164031 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:36.160127    2544 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-211.ec2.internal.18d461930b94ed10  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-211.ec2.internal,UID:ip-10-0-142-211.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-211.ec2.internal,},FirstTimestamp:2026-09-11 21:30:36.155325712 +0000 UTC m=+0.402769232,LastTimestamp:2026-09-11 21:30:36.155325712 +0000 UTC m=+0.402769232,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-211.ec2.internal,}"
Sep 11 21:30:36.164181 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:36.164148    2544 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 21:30:36.170383 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.170360    2544 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 21:30:36.171029 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.171011    2544 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 21:30:36.171165 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:36.171147    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-211.ec2.internal\" not found"
Sep 11 21:30:36.171805 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.171733    2544 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 21:30:36.171805 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.171751    2544 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 21:30:36.171944 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.171834    2544 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 21:30:36.172085 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.172071    2544 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 21:30:36.172085 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.172085    2544 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 21:30:36.172407 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.172393    2544 factory.go:55] Registering systemd factory
Sep 11 21:30:36.172518 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.172502    2544 factory.go:223] Registration of the systemd container factory successfully
Sep 11 21:30:36.172726 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.172711    2544 factory.go:153] Registering CRI-O factory
Sep 11 21:30:36.172726 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.172725    2544 factory.go:223] Registration of the crio container factory successfully
Sep 11 21:30:36.172862 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.172773    2544 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 21:30:36.172862 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.172789    2544 factory.go:103] Registering Raw factory
Sep 11 21:30:36.172862 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.172804    2544 manager.go:1196] Started watching for new ooms in manager
Sep 11 21:30:36.173210 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.173184    2544 manager.go:349] Starting recovery of all containers
Sep 11 21:30:36.182969 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:36.182851    2544 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-211.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 21:30:36.182969 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:36.182854    2544 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 21:30:36.183372 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.183347    2544 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2nbnj"
Sep 11 21:30:36.184889 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.184872    2544 manager.go:354] Recovery completed
Sep 11 21:30:36.189925 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.189908    2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:30:36.191826 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.191809    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:30:36.191889 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.191837    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:30:36.191889 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.191848    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:30:36.192442 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.192426    2544 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 21:30:36.192442 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.192440    2544 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 21:30:36.192567 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.192452    2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:30:36.192567 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.192464    2544 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 21:30:36.193681 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.193660    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:30:36.193769 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.193701    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:30:36.193769 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.193716    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:30:36.193930 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.193909    2544 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2nbnj"
Sep 11 21:30:36.195572 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.195553    2544 policy_none.go:49] "None policy: Start"
Sep 11 21:30:36.195664 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.195576    2544 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 21:30:36.195664 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.195593    2544 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 21:30:36.197937 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.197922    2544 policy_none.go:47] "Start"
Sep 11 21:30:36.230952 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.230757    2544 manager.go:337] "Starting Device Plugin manager"
Sep 11 21:30:36.253170 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:36.230983    2544 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 21:30:36.253170 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.230995    2544 server.go:87] "Starting device plugin registration server"
Sep 11 21:30:36.253170 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.231398    2544 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 21:30:36.253170 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.231412    2544 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 21:30:36.253170 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.231489    2544 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 21:30:36.253170 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.231589    2544 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 21:30:36.253170 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.231599    2544 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 21:30:36.253170 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:36.232242    2544 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 21:30:36.253170 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:36.232282    2544 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-211.ec2.internal\" not found"
Sep 11 21:30:36.306262 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.306229    2544 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 21:30:36.307463 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.307437    2544 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 21:30:36.307577 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.307486    2544 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 21:30:36.307577 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.307509    2544 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 21:30:36.307672 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:36.307618    2544 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 21:30:36.310167 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.310143    2544 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:30:36.332557 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.332476    2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:30:36.333535 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.333520    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:30:36.333621 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.333548    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:30:36.333621 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.333560    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:30:36.333621 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.333583    2544 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-211.ec2.internal"
Sep 11 21:30:36.345493 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.345478    2544 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-211.ec2.internal"
Sep 11 21:30:36.345539 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:36.345502    2544 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-211.ec2.internal\": node \"ip-10-0-142-211.ec2.internal\" not found"
Sep 11 21:30:36.374056 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:36.374029    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-211.ec2.internal\" not found"
Sep 11 21:30:36.408317 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.408289    2544 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-211.ec2.internal"]
Sep 11 21:30:36.408402 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.408368    2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:30:36.409368 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.409352    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:30:36.409457 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.409386    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:30:36.409457 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.409400    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:30:36.411578 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.411562    2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:30:36.411716 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.411701    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal"
Sep 11 21:30:36.411781 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.411742    2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:30:36.412368 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.412352    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:30:36.412437 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.412380    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:30:36.412437 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.412392    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:30:36.413126 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.413112    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:30:36.413171 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.413143    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:30:36.413171 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.413157    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:30:36.415105 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.415087    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-211.ec2.internal"
Sep 11 21:30:36.415171 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.415121    2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:30:36.415781 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.415765    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:30:36.415781 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.415789    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:30:36.415896 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.415798    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:30:36.441154 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:36.441127    2544 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-211.ec2.internal\" not found" node="ip-10-0-142-211.ec2.internal"
Sep 11 21:30:36.445554 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:36.445537    2544 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-211.ec2.internal\" not found" node="ip-10-0-142-211.ec2.internal"
Sep 11 21:30:36.473470 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.473437    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7e17897ec24162ac27d9b7d9ee876183-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal\" (UID: \"7e17897ec24162ac27d9b7d9ee876183\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal"
Sep 11 21:30:36.473470 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.473465    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e17897ec24162ac27d9b7d9ee876183-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal\" (UID: \"7e17897ec24162ac27d9b7d9ee876183\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal"
Sep 11 21:30:36.473676 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:36.473497    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/400563a539e706696660090857d9ffd0-config\") pod \"kube-apiserver-proxy-ip-10-0-142-211.ec2.internal\" (UID: \"400563a539e706696660090857d9ffd0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-211.ec2.internal"
Sep 11 21:30:36.474461 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:36.474443    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-211.ec2.internal\" not found"
Sep 11 21:30:37.852330 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:37.852301    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7e17897ec24162ac27d9b7d9ee876183-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal\" (UID: \"7e17897ec24162ac27d9b7d9ee876183\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal"
Sep 11 21:30:37.852330 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:37.852332    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e17897ec24162ac27d9b7d9ee876183-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal\" (UID: \"7e17897ec24162ac27d9b7d9ee876183\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal"
Sep 11 21:30:37.852496 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:37.852351    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/400563a539e706696660090857d9ffd0-config\") pod \"kube-apiserver-proxy-ip-10-0-142-211.ec2.internal\" (UID: \"400563a539e706696660090857d9ffd0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-211.ec2.internal"
Sep 11 21:30:37.852496 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:37.852403    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/400563a539e706696660090857d9ffd0-config\") pod \"kube-apiserver-proxy-ip-10-0-142-211.ec2.internal\" (UID: \"400563a539e706696660090857d9ffd0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-211.ec2.internal"
Sep 11 21:30:37.852496 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:37.852411    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7e17897ec24162ac27d9b7d9ee876183-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal\" (UID: \"7e17897ec24162ac27d9b7d9ee876183\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal"
Sep 11 21:30:37.852496 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:37.852425    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e17897ec24162ac27d9b7d9ee876183-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal\" (UID: \"7e17897ec24162ac27d9b7d9ee876183\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal"
Sep 11 21:30:37.853343 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:37.853327    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-211.ec2.internal\" not found"
Sep 11 21:30:37.953462 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:37.953373    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-211.ec2.internal\" not found"
Sep 11 21:30:38.023583 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:38.023554    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal"
Sep 11 21:30:38.027645 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:38.027621    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-211.ec2.internal"
Sep 11 21:30:38.053919 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:38.053887    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-211.ec2.internal\" not found"
Sep 11 21:30:38.154396 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:38.154359    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-211.ec2.internal\" not found"
Sep 11 21:30:38.254820 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:38.254737    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-211.ec2.internal\" not found"
Sep 11 21:30:38.355236 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:38.355194    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-211.ec2.internal\" not found"
Sep 11 21:30:38.376372 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:38.376347    2544 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 21:30:38.376517 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:38.376477    2544 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 21:30:38.455609 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:38.455578    2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-211.ec2.internal\" not found"
Sep 11 21:30:38.477613 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:38.477577    2544 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 21:25:37 +0000 UTC" deadline="2028-05-05 02:21:17.110728006 +0000 UTC"
Sep 11 21:30:38.477613 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:38.477613    2544 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14428h50m38.633120654s"
Sep 11 21:30:38.529584 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:38.529512    2544 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:30:38.550213 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:38.550177    2544 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal"
Sep 11 21:30:38.584358 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:38.584326    2544 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 21:30:38.586565 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:38.586544    2544 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-211.ec2.internal"
Sep 11 21:30:38.609850 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:38.609822    2544 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 21:30:38.641419 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:38.641387    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400563a539e706696660090857d9ffd0.slice/crio-f68915010a606c08de83f3cff9120ea26a8a8036b72f675fd85ccd69aab7fb64 WatchSource:0}: Error finding container f68915010a606c08de83f3cff9120ea26a8a8036b72f675fd85ccd69aab7fb64: Status 404 returned error can't find the container with id f68915010a606c08de83f3cff9120ea26a8a8036b72f675fd85ccd69aab7fb64
Sep 11 21:30:38.647147 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:38.647130    2544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:30:38.848470 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:38.848383    2544 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:30:38.901160 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:38.901125    2544 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:30:39.420514 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.420481    2544 apiserver.go:52] "Watching apiserver"
Sep 11 21:30:39.428530 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.428500    2544 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 21:30:39.428999 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.428971    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7","openshift-image-registry/node-ca-sh5wj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal","openshift-multus/multus-additional-cni-plugins-rljhw","kube-system/global-pull-secret-syncer-fjvb6","openshift-cluster-node-tuning-operator/tuned-z8wtk","openshift-dns/node-resolver-6gf57","openshift-multus/multus-lfxpn","openshift-multus/network-metrics-daemon-k94pz","openshift-network-diagnostics/network-check-target-wmjnx","openshift-network-operator/iptables-alerter-qrcsr","openshift-ovn-kubernetes/ovnkube-node-5d2lj","kube-system/konnectivity-agent-dkd6r","kube-system/kube-apiserver-proxy-ip-10-0-142-211.ec2.internal"]
Sep 11 21:30:39.431536 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.431504    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6gf57"
Sep 11 21:30:39.433792 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.433641    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dkd6r"
Sep 11 21:30:39.434968 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.434146    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 21:30:39.434968 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.434540    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rdv4w\""
Sep 11 21:30:39.435387 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.435336    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 21:30:39.436163 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.435612    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 21:30:39.436163 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.435897    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.436407 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.436387    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5s2gr\""
Sep 11 21:30:39.438986 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.438126    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.438986 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.438394    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 21:30:39.438986 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.438667    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 21:30:39.438986 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.438685    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 21:30:39.438986 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.438763    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 21:30:39.438986 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.438851    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fbdtq\""
Sep 11 21:30:39.438986 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.438948    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 21:30:39.440408 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.440304    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 21:30:39.440525 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.440466    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rgmgg\""
Sep 11 21:30:39.440682 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.440664    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 21:30:39.440747 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.440709    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:39.440857 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:39.440766    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjvb6" podUID="fef5eac9-cf6d-40b0-9bb2-374516793dc2"
Sep 11 21:30:39.442729 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.442710    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.445178 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.445133    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 21:30:39.446507 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.446407    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sx64x\""
Sep 11 21:30:39.449071 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.446658    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:30:39.449071 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.446929    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:39.449071 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:39.447023    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k94pz" podUID="047e6301-db2b-4c57-86ad-20ca80ff019a"
Sep 11 21:30:39.449071 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.447348    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.449071 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.448669    2544 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 21:30:39.450491 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.450027    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sh5wj"
Sep 11 21:30:39.450491 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.450251    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 21:30:39.450491 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.450290    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 21:30:39.452097 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.452061    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tbpbt\""
Sep 11 21:30:39.452243 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.452209    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 21:30:39.452667 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.452648    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 21:30:39.453309 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.453035    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 21:30:39.453309 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.453077    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:30:39.453309 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.453122    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 21:30:39.453309 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:39.453150    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmjnx" podUID="6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b"
Sep 11 21:30:39.453535 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.453425    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mdkhs\""
Sep 11 21:30:39.455416 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.455396    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qrcsr"
Sep 11 21:30:39.457921 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.457900    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.459233 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.459198    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:30:39.459317 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.459266    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5j6pv\""
Sep 11 21:30:39.459359 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.459315    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 21:30:39.459498 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.459481    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 21:30:39.460351 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.460325    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 21:30:39.460438 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.460399    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 21:30:39.460651 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.460633    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n2fcl\""
Sep 11 21:30:39.460876 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.460854    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/728ca319-dbd4-4ade-8a93-10191019f4df-serviceca\") pod \"node-ca-sh5wj\" (UID: \"728ca319-dbd4-4ade-8a93-10191019f4df\") " pod="openshift-image-registry/node-ca-sh5wj"
Sep 11 21:30:39.461033 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461009    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-sysctl-d\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.461170 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461043    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-systemd\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.461170 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461067    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/728ca319-dbd4-4ade-8a93-10191019f4df-host\") pod \"node-ca-sh5wj\" (UID: \"728ca319-dbd4-4ade-8a93-10191019f4df\") " pod="openshift-image-registry/node-ca-sh5wj"
Sep 11 21:30:39.461170 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461090    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fec47591-6ba3-441d-8541-ee68a0470a2b-agent-certs\") pod \"konnectivity-agent-dkd6r\" (UID: \"fec47591-6ba3-441d-8541-ee68a0470a2b\") " pod="kube-system/konnectivity-agent-dkd6r"
Sep 11 21:30:39.461170 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461116    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-host-run-multus-certs\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.461528 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461175    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/07c49e47-3378-4c6c-984a-802481fbf22c-hosts-file\") pod \"node-resolver-6gf57\" (UID: \"07c49e47-3378-4c6c-984a-802481fbf22c\") " pod="openshift-dns/node-resolver-6gf57"
Sep 11 21:30:39.461528 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461244    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fef5eac9-cf6d-40b0-9bb2-374516793dc2-kubelet-config\") pod \"global-pull-secret-syncer-fjvb6\" (UID: \"fef5eac9-cf6d-40b0-9bb2-374516793dc2\") " pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:39.461528 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461290    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-system-cni-dir\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.461528 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461331    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b6n7\" (UniqueName: \"kubernetes.io/projected/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-kube-api-access-6b6n7\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.461528 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461364    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c44d4afa-8d65-4feb-83f9-7f6ed664511a-system-cni-dir\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.461528 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461393    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret\") pod \"global-pull-secret-syncer-fjvb6\" (UID: \"fef5eac9-cf6d-40b0-9bb2-374516793dc2\") " pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:39.461528 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461436    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-etc-kubernetes\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.461528 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461467    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c44d4afa-8d65-4feb-83f9-7f6ed664511a-cnibin\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.461528 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461500    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9c045b9b-c992-4a35-ba9d-0769d9e22dae-etc-selinux\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.461528 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461524    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl6th\" (UniqueName: \"kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th\") pod \"network-check-target-wmjnx\" (UID: \"6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b\") " pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:30:39.462027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461563    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c44d4afa-8d65-4feb-83f9-7f6ed664511a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.462027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461567    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 21:30:39.462027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461584    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9c045b9b-c992-4a35-ba9d-0769d9e22dae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.462027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461608    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-sysconfig\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.462027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461634    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-sysctl-conf\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.462027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461671    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c44d4afa-8d65-4feb-83f9-7f6ed664511a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.462027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461694    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbgjz\" (UniqueName: \"kubernetes.io/projected/728ca319-dbd4-4ade-8a93-10191019f4df-kube-api-access-jbgjz\") pod \"node-ca-sh5wj\" (UID: \"728ca319-dbd4-4ade-8a93-10191019f4df\") " pod="openshift-image-registry/node-ca-sh5wj"
Sep 11 21:30:39.462027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461725    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-multus-cni-dir\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.462027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461743    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-host-run-k8s-cni-cncf-io\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.462027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461767    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fec47591-6ba3-441d-8541-ee68a0470a2b-konnectivity-ca\") pod \"konnectivity-agent-dkd6r\" (UID: \"fec47591-6ba3-441d-8541-ee68a0470a2b\") " pod="kube-system/konnectivity-agent-dkd6r"
Sep 11 21:30:39.462027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461789    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 21:30:39.462027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461789    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-host-run-netns\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.462027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461876    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-host-var-lib-kubelet\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.462027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461916    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-lib-modules\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.462027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461947    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-host\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.462027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.461982    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c44d4afa-8d65-4feb-83f9-7f6ed664511a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.462027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462014    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-cnibin\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.463072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462045    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-host-var-lib-cni-bin\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.463072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462072    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs\") pod \"network-metrics-daemon-k94pz\" (UID: \"047e6301-db2b-4c57-86ad-20ca80ff019a\") " pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:39.463072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462096    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9c045b9b-c992-4a35-ba9d-0769d9e22dae-sys-fs\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.463072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462125    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-multus-socket-dir-parent\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.463072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462167    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-kubernetes\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.463072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462183    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-tuned\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.463072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462198    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c49e47-3378-4c6c-984a-802481fbf22c-tmp-dir\") pod \"node-resolver-6gf57\" (UID: \"07c49e47-3378-4c6c-984a-802481fbf22c\") " pod="openshift-dns/node-resolver-6gf57"
Sep 11 21:30:39.463072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462212    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pz84\" (UniqueName: \"kubernetes.io/projected/c44d4afa-8d65-4feb-83f9-7f6ed664511a-kube-api-access-7pz84\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.463072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462280    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fef5eac9-cf6d-40b0-9bb2-374516793dc2-dbus\") pod \"global-pull-secret-syncer-fjvb6\" (UID: \"fef5eac9-cf6d-40b0-9bb2-374516793dc2\") " pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:39.463072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462303    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9c045b9b-c992-4a35-ba9d-0769d9e22dae-socket-dir\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.463072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462326    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9c045b9b-c992-4a35-ba9d-0769d9e22dae-device-dir\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.463072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462360    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-hostroot\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.463072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462418    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-run\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.463072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462440    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-var-lib-kubelet\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.463072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462463    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-649wv\" (UniqueName: \"kubernetes.io/projected/9c045b9b-c992-4a35-ba9d-0769d9e22dae-kube-api-access-649wv\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.463072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462480    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-multus-conf-dir\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.463971 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462520    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cecafdee-703e-498c-91ad-573fa4e60d2d-multus-daemon-config\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.463971 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462570    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-sys\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.463971 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462598    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9c045b9b-c992-4a35-ba9d-0769d9e22dae-registration-dir\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.463971 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462624    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skvrl\" (UniqueName: \"kubernetes.io/projected/cecafdee-703e-498c-91ad-573fa4e60d2d-kube-api-access-skvrl\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.463971 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462646    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-tmp\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.463971 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462673    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pz4c\" (UniqueName: \"kubernetes.io/projected/047e6301-db2b-4c57-86ad-20ca80ff019a-kube-api-access-2pz4c\") pod \"network-metrics-daemon-k94pz\" (UID: \"047e6301-db2b-4c57-86ad-20ca80ff019a\") " pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:39.463971 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462696    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cecafdee-703e-498c-91ad-573fa4e60d2d-cni-binary-copy\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.463971 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462720    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-host-var-lib-cni-multus\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.463971 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462743    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-modprobe-d\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.463971 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462785    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwgkf\" (UniqueName: \"kubernetes.io/projected/07c49e47-3378-4c6c-984a-802481fbf22c-kube-api-access-kwgkf\") pod \"node-resolver-6gf57\" (UID: \"07c49e47-3378-4c6c-984a-802481fbf22c\") " pod="openshift-dns/node-resolver-6gf57"
Sep 11 21:30:39.463971 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.462848    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c44d4afa-8d65-4feb-83f9-7f6ed664511a-cni-binary-copy\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.464778 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.464125    2544 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 21:30:39.464778 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.464176    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 21:30:39.464778 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.464346    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 21:30:39.496639 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.496614    2544 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gmxlf"
Sep 11 21:30:39.509667 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.509639    2544 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gmxlf"
Sep 11 21:30:39.551058 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.551022    2544 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 21:30:39.564063 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564032    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-ovnkube-script-lib\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.564253 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564093    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwgkf\" (UniqueName: \"kubernetes.io/projected/07c49e47-3378-4c6c-984a-802481fbf22c-kube-api-access-kwgkf\") pod \"node-resolver-6gf57\" (UID: \"07c49e47-3378-4c6c-984a-802481fbf22c\") " pod="openshift-dns/node-resolver-6gf57"
Sep 11 21:30:39.564253 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564118    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c44d4afa-8d65-4feb-83f9-7f6ed664511a-cni-binary-copy\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.564253 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564141    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/728ca319-dbd4-4ade-8a93-10191019f4df-serviceca\") pod \"node-ca-sh5wj\" (UID: \"728ca319-dbd4-4ade-8a93-10191019f4df\") " pod="openshift-image-registry/node-ca-sh5wj"
Sep 11 21:30:39.564253 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564163    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-sysctl-d\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.564460 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564288    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-systemd\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.564460 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564313    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-slash\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.564460 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564334    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-etc-openvswitch\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.564460 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564356    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/728ca319-dbd4-4ade-8a93-10191019f4df-host\") pod \"node-ca-sh5wj\" (UID: \"728ca319-dbd4-4ade-8a93-10191019f4df\") " pod="openshift-image-registry/node-ca-sh5wj"
Sep 11 21:30:39.564460 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564385    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fec47591-6ba3-441d-8541-ee68a0470a2b-agent-certs\") pod \"konnectivity-agent-dkd6r\" (UID: \"fec47591-6ba3-441d-8541-ee68a0470a2b\") " pod="kube-system/konnectivity-agent-dkd6r"
Sep 11 21:30:39.564460 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564402    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-systemd\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.564460 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564406    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-host-run-multus-certs\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.564460 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564434    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-sysctl-d\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.564460 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564453    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-host-run-multus-certs\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.564909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564469    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/07c49e47-3378-4c6c-984a-802481fbf22c-hosts-file\") pod \"node-resolver-6gf57\" (UID: \"07c49e47-3378-4c6c-984a-802481fbf22c\") " pod="openshift-dns/node-resolver-6gf57"
Sep 11 21:30:39.564909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564488    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/728ca319-dbd4-4ade-8a93-10191019f4df-host\") pod \"node-ca-sh5wj\" (UID: \"728ca319-dbd4-4ade-8a93-10191019f4df\") " pod="openshift-image-registry/node-ca-sh5wj"
Sep 11 21:30:39.564909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564497    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fef5eac9-cf6d-40b0-9bb2-374516793dc2-kubelet-config\") pod \"global-pull-secret-syncer-fjvb6\" (UID: \"fef5eac9-cf6d-40b0-9bb2-374516793dc2\") " pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:39.564909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564521    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-system-cni-dir\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.564909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564542    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/07c49e47-3378-4c6c-984a-802481fbf22c-hosts-file\") pod \"node-resolver-6gf57\" (UID: \"07c49e47-3378-4c6c-984a-802481fbf22c\") " pod="openshift-dns/node-resolver-6gf57"
Sep 11 21:30:39.564909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564576    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fef5eac9-cf6d-40b0-9bb2-374516793dc2-kubelet-config\") pod \"global-pull-secret-syncer-fjvb6\" (UID: \"fef5eac9-cf6d-40b0-9bb2-374516793dc2\") " pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:39.564909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564544    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b6n7\" (UniqueName: \"kubernetes.io/projected/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-kube-api-access-6b6n7\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.564909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564641    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c44d4afa-8d65-4feb-83f9-7f6ed664511a-system-cni-dir\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.564909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564659    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-system-cni-dir\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.564909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564695    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret\") pod \"global-pull-secret-syncer-fjvb6\" (UID: \"fef5eac9-cf6d-40b0-9bb2-374516793dc2\") " pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:39.564909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564727    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-etc-kubernetes\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.564909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564778    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttqqz\" (UniqueName: \"kubernetes.io/projected/8ba240ac-88f7-4d2c-80c2-4567bb23184b-kube-api-access-ttqqz\") pod \"iptables-alerter-qrcsr\" (UID: \"8ba240ac-88f7-4d2c-80c2-4567bb23184b\") " pod="openshift-network-operator/iptables-alerter-qrcsr"
Sep 11 21:30:39.564909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564808    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c44d4afa-8d65-4feb-83f9-7f6ed664511a-cni-binary-copy\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.564909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564818    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/728ca319-dbd4-4ade-8a93-10191019f4df-serviceca\") pod \"node-ca-sh5wj\" (UID: \"728ca319-dbd4-4ade-8a93-10191019f4df\") " pod="openshift-image-registry/node-ca-sh5wj"
Sep 11 21:30:39.564909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564831    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c44d4afa-8d65-4feb-83f9-7f6ed664511a-cnibin\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.564909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564824    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c44d4afa-8d65-4feb-83f9-7f6ed664511a-system-cni-dir\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.564909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564874    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9c045b9b-c992-4a35-ba9d-0769d9e22dae-etc-selinux\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.565592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564878    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c44d4afa-8d65-4feb-83f9-7f6ed664511a-cnibin\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.565592 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:39.564831    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:39.565592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564883    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-etc-kubernetes\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.565592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564946    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl6th\" (UniqueName: \"kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th\") pod \"network-check-target-wmjnx\" (UID: \"6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b\") " pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:30:39.565592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.564979    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.565592 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:39.565023    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret podName:fef5eac9-cf6d-40b0-9bb2-374516793dc2 nodeName:}" failed. No retries permitted until 2026-09-11 21:30:40.064998614 +0000 UTC m=+3.034415426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret") pod "global-pull-secret-syncer-fjvb6" (UID: "fef5eac9-cf6d-40b0-9bb2-374516793dc2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:39.565592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565022    2544 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 21:30:39.565592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565534    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c44d4afa-8d65-4feb-83f9-7f6ed664511a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.565592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565546    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9c045b9b-c992-4a35-ba9d-0769d9e22dae-etc-selinux\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.565592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565570    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9c045b9b-c992-4a35-ba9d-0769d9e22dae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.565990 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565602    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-sysconfig\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.565990 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565618    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-sysctl-conf\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.565990 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565641    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c44d4afa-8d65-4feb-83f9-7f6ed664511a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.565990 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565657    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbgjz\" (UniqueName: \"kubernetes.io/projected/728ca319-dbd4-4ade-8a93-10191019f4df-kube-api-access-jbgjz\") pod \"node-ca-sh5wj\" (UID: \"728ca319-dbd4-4ade-8a93-10191019f4df\") " pod="openshift-image-registry/node-ca-sh5wj"
Sep 11 21:30:39.565990 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565673    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-multus-cni-dir\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.565990 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565700    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-host-run-k8s-cni-cncf-io\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.565990 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565721    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fec47591-6ba3-441d-8541-ee68a0470a2b-konnectivity-ca\") pod \"konnectivity-agent-dkd6r\" (UID: \"fec47591-6ba3-441d-8541-ee68a0470a2b\") " pod="kube-system/konnectivity-agent-dkd6r"
Sep 11 21:30:39.565990 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565744    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-host-run-netns\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.565990 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565767    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-multus-cni-dir\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.565990 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565761    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9c045b9b-c992-4a35-ba9d-0769d9e22dae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.565990 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565796    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-sysconfig\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.565990 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565863    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-sysctl-conf\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.565990 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565943    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c44d4afa-8d65-4feb-83f9-7f6ed664511a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.565990 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565934    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-host-var-lib-kubelet\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.565990 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.565967    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-host-run-k8s-cni-cncf-io\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.566728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566014    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-host-run-netns\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.566728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566017    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-host-var-lib-kubelet\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.566728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566038    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-lib-modules\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.566728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566064    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-host\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.566728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566093    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c44d4afa-8d65-4feb-83f9-7f6ed664511a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.566728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566148    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-host\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.566728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566174    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-lib-modules\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.566728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566178    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ba240ac-88f7-4d2c-80c2-4567bb23184b-iptables-alerter-script\") pod \"iptables-alerter-qrcsr\" (UID: \"8ba240ac-88f7-4d2c-80c2-4567bb23184b\") " pod="openshift-network-operator/iptables-alerter-qrcsr"
Sep 11 21:30:39.566728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566236    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-var-lib-openvswitch\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.566728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566267    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-cnibin\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.566728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566297    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-host-var-lib-cni-bin\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.566728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566322    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ba240ac-88f7-4d2c-80c2-4567bb23184b-host-slash\") pod \"iptables-alerter-qrcsr\" (UID: \"8ba240ac-88f7-4d2c-80c2-4567bb23184b\") " pod="openshift-network-operator/iptables-alerter-qrcsr"
Sep 11 21:30:39.566728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566342    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-host-var-lib-cni-bin\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.566728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566614    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-cnibin\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.566728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566664    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-node-log\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.566728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566725    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs\") pod \"network-metrics-daemon-k94pz\" (UID: \"047e6301-db2b-4c57-86ad-20ca80ff019a\") " pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:39.567448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566799    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fec47591-6ba3-441d-8541-ee68a0470a2b-konnectivity-ca\") pod \"konnectivity-agent-dkd6r\" (UID: \"fec47591-6ba3-441d-8541-ee68a0470a2b\") " pod="kube-system/konnectivity-agent-dkd6r"
Sep 11 21:30:39.567448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566819    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9c045b9b-c992-4a35-ba9d-0769d9e22dae-sys-fs\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.567448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566826    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c44d4afa-8d65-4feb-83f9-7f6ed664511a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.567448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566854    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-multus-socket-dir-parent\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.567448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566914    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-kubernetes\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.567448 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:39.566951    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:39.567448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566958    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-tuned\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.567448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.566996    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-run-ovn\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.567448 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:39.567018    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs podName:047e6301-db2b-4c57-86ad-20ca80ff019a nodeName:}" failed. No retries permitted until 2026-09-11 21:30:40.066999546 +0000 UTC m=+3.036416356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs") pod "network-metrics-daemon-k94pz" (UID: "047e6301-db2b-4c57-86ad-20ca80ff019a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:39.567448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567018    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c44d4afa-8d65-4feb-83f9-7f6ed664511a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.567448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567095    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-kubernetes\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.567448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567106    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9c045b9b-c992-4a35-ba9d-0769d9e22dae-sys-fs\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.567448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567122    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-multus-socket-dir-parent\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.567448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567182    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.567448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567230    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpftv\" (UniqueName: \"kubernetes.io/projected/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-kube-api-access-kpftv\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.567448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567272    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c49e47-3378-4c6c-984a-802481fbf22c-tmp-dir\") pod \"node-resolver-6gf57\" (UID: \"07c49e47-3378-4c6c-984a-802481fbf22c\") " pod="openshift-dns/node-resolver-6gf57"
Sep 11 21:30:39.567448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567327    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pz84\" (UniqueName: \"kubernetes.io/projected/c44d4afa-8d65-4feb-83f9-7f6ed664511a-kube-api-access-7pz84\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.568340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567378    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fef5eac9-cf6d-40b0-9bb2-374516793dc2-dbus\") pod \"global-pull-secret-syncer-fjvb6\" (UID: \"fef5eac9-cf6d-40b0-9bb2-374516793dc2\") " pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:39.568340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567437    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9c045b9b-c992-4a35-ba9d-0769d9e22dae-socket-dir\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.568340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567469    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9c045b9b-c992-4a35-ba9d-0769d9e22dae-device-dir\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.568340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567503    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-hostroot\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.568340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567534    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-run\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.568340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567560    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-var-lib-kubelet\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.568340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567582    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9c045b9b-c992-4a35-ba9d-0769d9e22dae-socket-dir\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.568340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567594    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-649wv\" (UniqueName: \"kubernetes.io/projected/9c045b9b-c992-4a35-ba9d-0769d9e22dae-kube-api-access-649wv\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.568340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567627    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-multus-conf-dir\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.568340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567662    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cecafdee-703e-498c-91ad-573fa4e60d2d-multus-daemon-config\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.568340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567663    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fef5eac9-cf6d-40b0-9bb2-374516793dc2-dbus\") pod \"global-pull-secret-syncer-fjvb6\" (UID: \"fef5eac9-cf6d-40b0-9bb2-374516793dc2\") " pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:39.568340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567694    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-sys\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.568340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567706    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c49e47-3378-4c6c-984a-802481fbf22c-tmp-dir\") pod \"node-resolver-6gf57\" (UID: \"07c49e47-3378-4c6c-984a-802481fbf22c\") " pod="openshift-dns/node-resolver-6gf57"
Sep 11 21:30:39.568340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567724    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-log-socket\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.568340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567739    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-run\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.568340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567763    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-cni-bin\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.568340 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567795    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-env-overrides\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.571159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567832    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9c045b9b-c992-4a35-ba9d-0769d9e22dae-registration-dir\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.571159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567874    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skvrl\" (UniqueName: \"kubernetes.io/projected/cecafdee-703e-498c-91ad-573fa4e60d2d-kube-api-access-skvrl\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.571159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567906    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-sys\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.571159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567923    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-tmp\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.571159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567954    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-cni-netd\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.571159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567967    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-var-lib-kubelet\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.571159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567986    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-ovnkube-config\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.571159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.568016    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-systemd-units\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.571159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.568042    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-run-netns\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.571159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.568065    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-hostroot\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.571159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.568075    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-run-systemd\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.571159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.568110    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-ovn-node-metrics-cert\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.571159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.568148    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pz4c\" (UniqueName: \"kubernetes.io/projected/047e6301-db2b-4c57-86ad-20ca80ff019a-kube-api-access-2pz4c\") pod \"network-metrics-daemon-k94pz\" (UID: \"047e6301-db2b-4c57-86ad-20ca80ff019a\") " pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:39.571159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.568187    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cecafdee-703e-498c-91ad-573fa4e60d2d-cni-binary-copy\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.571159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.568240    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-host-var-lib-cni-multus\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.571159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.568281    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-modprobe-d\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.571159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.567834    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9c045b9b-c992-4a35-ba9d-0769d9e22dae-device-dir\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.572003 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.568315    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-kubelet\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.572003 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.568420    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-run-openvswitch\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.572003 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.569015    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cecafdee-703e-498c-91ad-573fa4e60d2d-multus-daemon-config\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.572003 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.569115    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-host-var-lib-cni-multus\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.572003 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.569194    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cecafdee-703e-498c-91ad-573fa4e60d2d-cni-binary-copy\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.572003 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.569282    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-modprobe-d\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.572003 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.568422    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cecafdee-703e-498c-91ad-573fa4e60d2d-multus-conf-dir\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.572003 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.569558    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fec47591-6ba3-441d-8541-ee68a0470a2b-agent-certs\") pod \"konnectivity-agent-dkd6r\" (UID: \"fec47591-6ba3-441d-8541-ee68a0470a2b\") " pod="kube-system/konnectivity-agent-dkd6r"
Sep 11 21:30:39.572003 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.570890    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-tmp\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.572003 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.570965    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-etc-tuned\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.572003 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.571078    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9c045b9b-c992-4a35-ba9d-0769d9e22dae-registration-dir\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.577251 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:39.577232    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:30:39.577354 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:39.577255    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:30:39.577354 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:39.577265    2544 projected.go:196] Error preparing data for projected volume kube-api-access-zl6th for pod openshift-network-diagnostics/network-check-target-wmjnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:39.577354 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:39.577317    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th podName:6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b nodeName:}" failed. No retries permitted until 2026-09-11 21:30:40.077304613 +0000 UTC m=+3.046721411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zl6th" (UniqueName: "kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th") pod "network-check-target-wmjnx" (UID: "6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:39.582842 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.582821    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pz4c\" (UniqueName: \"kubernetes.io/projected/047e6301-db2b-4c57-86ad-20ca80ff019a-kube-api-access-2pz4c\") pod \"network-metrics-daemon-k94pz\" (UID: \"047e6301-db2b-4c57-86ad-20ca80ff019a\") " pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:39.583280 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.583260    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwgkf\" (UniqueName: \"kubernetes.io/projected/07c49e47-3378-4c6c-984a-802481fbf22c-kube-api-access-kwgkf\") pod \"node-resolver-6gf57\" (UID: \"07c49e47-3378-4c6c-984a-802481fbf22c\") " pod="openshift-dns/node-resolver-6gf57"
Sep 11 21:30:39.583280 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.583271    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pz84\" (UniqueName: \"kubernetes.io/projected/c44d4afa-8d65-4feb-83f9-7f6ed664511a-kube-api-access-7pz84\") pod \"multus-additional-cni-plugins-rljhw\" (UID: \"c44d4afa-8d65-4feb-83f9-7f6ed664511a\") " pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.583420 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.583366    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbgjz\" (UniqueName: \"kubernetes.io/projected/728ca319-dbd4-4ade-8a93-10191019f4df-kube-api-access-jbgjz\") pod \"node-ca-sh5wj\" (UID: \"728ca319-dbd4-4ade-8a93-10191019f4df\") " pod="openshift-image-registry/node-ca-sh5wj"
Sep 11 21:30:39.583905 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.583878    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-649wv\" (UniqueName: \"kubernetes.io/projected/9c045b9b-c992-4a35-ba9d-0769d9e22dae-kube-api-access-649wv\") pod \"aws-ebs-csi-driver-node-ng8h7\" (UID: \"9c045b9b-c992-4a35-ba9d-0769d9e22dae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.585325 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.585304    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skvrl\" (UniqueName: \"kubernetes.io/projected/cecafdee-703e-498c-91ad-573fa4e60d2d-kube-api-access-skvrl\") pod \"multus-lfxpn\" (UID: \"cecafdee-703e-498c-91ad-573fa4e60d2d\") " pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.588675 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.588655    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b6n7\" (UniqueName: \"kubernetes.io/projected/5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e-kube-api-access-6b6n7\") pod \"tuned-z8wtk\" (UID: \"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e\") " pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.590391 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.590334    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal" event={"ID":"7e17897ec24162ac27d9b7d9ee876183","Type":"ContainerStarted","Data":"bfe4223c69c128bae447fefe92df77997d57d8efe9b216650420ac4e90ed1cd1"}
Sep 11 21:30:39.591676 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.591647    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-211.ec2.internal" event={"ID":"400563a539e706696660090857d9ffd0","Type":"ContainerStarted","Data":"f68915010a606c08de83f3cff9120ea26a8a8036b72f675fd85ccd69aab7fb64"}
Sep 11 21:30:39.668884 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.668849    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ba240ac-88f7-4d2c-80c2-4567bb23184b-iptables-alerter-script\") pod \"iptables-alerter-qrcsr\" (UID: \"8ba240ac-88f7-4d2c-80c2-4567bb23184b\") " pod="openshift-network-operator/iptables-alerter-qrcsr"
Sep 11 21:30:39.668884 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.668887    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-var-lib-openvswitch\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669134 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.668913    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ba240ac-88f7-4d2c-80c2-4567bb23184b-host-slash\") pod \"iptables-alerter-qrcsr\" (UID: \"8ba240ac-88f7-4d2c-80c2-4567bb23184b\") " pod="openshift-network-operator/iptables-alerter-qrcsr"
Sep 11 21:30:39.669134 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.668965    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ba240ac-88f7-4d2c-80c2-4567bb23184b-host-slash\") pod \"iptables-alerter-qrcsr\" (UID: \"8ba240ac-88f7-4d2c-80c2-4567bb23184b\") " pod="openshift-network-operator/iptables-alerter-qrcsr"
Sep 11 21:30:39.669134 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.668975    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-var-lib-openvswitch\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669134 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.668992    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-node-log\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669134 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669020    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-run-ovn\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669134 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669034    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669134 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669050    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpftv\" (UniqueName: \"kubernetes.io/projected/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-kube-api-access-kpftv\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669134 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669084    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-log-socket\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669134 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669103    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-node-log\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669134 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669106    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-cni-bin\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669134 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669139    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-env-overrides\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669643 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669142    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669643 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669155    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-run-ovn\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669643 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669159    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-cni-netd\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669643 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669189    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-ovnkube-config\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669643 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669191    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-cni-bin\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669643 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669195    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-log-socket\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669643 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669240    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-systemd-units\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669643 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669326    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-run-netns\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669643 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669355    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-run-systemd\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669643 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669380    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-ovn-node-metrics-cert\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669643 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669430    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-run-systemd\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669643 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669463    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-cni-netd\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669643 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669489    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-run-netns\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669643 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669528    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ba240ac-88f7-4d2c-80c2-4567bb23184b-iptables-alerter-script\") pod \"iptables-alerter-qrcsr\" (UID: \"8ba240ac-88f7-4d2c-80c2-4567bb23184b\") " pod="openshift-network-operator/iptables-alerter-qrcsr"
Sep 11 21:30:39.669643 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669538    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-systemd-units\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.669643 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669617    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-kubelet\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.670394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669681    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-kubelet\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.670394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669706    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-run-openvswitch\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.670394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669747    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-ovnkube-script-lib\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.670394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669782    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-slash\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.670394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669772    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-ovnkube-config\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.670394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669829    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-etc-openvswitch\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.670394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669786    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-run-openvswitch\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.670394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669850    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-slash\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.670394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669866    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-env-overrides\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.670394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669898    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttqqz\" (UniqueName: \"kubernetes.io/projected/8ba240ac-88f7-4d2c-80c2-4567bb23184b-kube-api-access-ttqqz\") pod \"iptables-alerter-qrcsr\" (UID: \"8ba240ac-88f7-4d2c-80c2-4567bb23184b\") " pod="openshift-network-operator/iptables-alerter-qrcsr"
Sep 11 21:30:39.670394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669919    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-etc-openvswitch\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.670394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.669937    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.670394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.670023    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.670394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.670189    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-ovnkube-script-lib\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.671875 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.671817    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-ovn-node-metrics-cert\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.678149 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.678128    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttqqz\" (UniqueName: \"kubernetes.io/projected/8ba240ac-88f7-4d2c-80c2-4567bb23184b-kube-api-access-ttqqz\") pod \"iptables-alerter-qrcsr\" (UID: \"8ba240ac-88f7-4d2c-80c2-4567bb23184b\") " pod="openshift-network-operator/iptables-alerter-qrcsr"
Sep 11 21:30:39.679669 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.679646    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpftv\" (UniqueName: \"kubernetes.io/projected/5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd-kube-api-access-kpftv\") pod \"ovnkube-node-5d2lj\" (UID: \"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.749823 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.749782    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6gf57"
Sep 11 21:30:39.758530 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.758507    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dkd6r"
Sep 11 21:30:39.770316 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.770268    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lfxpn"
Sep 11 21:30:39.775747 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.775721    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rljhw"
Sep 11 21:30:39.783577 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.783558    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z8wtk"
Sep 11 21:30:39.791273 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.791247    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7"
Sep 11 21:30:39.797895 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.797873    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sh5wj"
Sep 11 21:30:39.807541 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.807516    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qrcsr"
Sep 11 21:30:39.814342 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.814318    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:30:39.901914 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:39.901880    2544 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:30:40.073072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:40.072990    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret\") pod \"global-pull-secret-syncer-fjvb6\" (UID: \"fef5eac9-cf6d-40b0-9bb2-374516793dc2\") " pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:40.073072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:40.073048    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs\") pod \"network-metrics-daemon-k94pz\" (UID: \"047e6301-db2b-4c57-86ad-20ca80ff019a\") " pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:40.073333 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:40.073171    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:40.073333 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:40.073173    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:40.073333 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:40.073286    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret podName:fef5eac9-cf6d-40b0-9bb2-374516793dc2 nodeName:}" failed. No retries permitted until 2026-09-11 21:30:41.073244931 +0000 UTC m=+4.042661747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret") pod "global-pull-secret-syncer-fjvb6" (UID: "fef5eac9-cf6d-40b0-9bb2-374516793dc2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:40.073469 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:40.073353    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs podName:047e6301-db2b-4c57-86ad-20ca80ff019a nodeName:}" failed. No retries permitted until 2026-09-11 21:30:41.073334354 +0000 UTC m=+4.042751153 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs") pod "network-metrics-daemon-k94pz" (UID: "047e6301-db2b-4c57-86ad-20ca80ff019a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:40.173437 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:40.173407    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl6th\" (UniqueName: \"kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th\") pod \"network-check-target-wmjnx\" (UID: \"6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b\") " pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:30:40.173569 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:40.173545    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:30:40.173569 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:40.173561    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:30:40.173658 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:40.173571    2544 projected.go:196] Error preparing data for projected volume kube-api-access-zl6th for pod openshift-network-diagnostics/network-check-target-wmjnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:40.173658 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:40.173618    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th podName:6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b nodeName:}" failed. No retries permitted until 2026-09-11 21:30:41.173606286 +0000 UTC m=+4.143023084 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zl6th" (UniqueName: "kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th") pod "network-check-target-wmjnx" (UID: "6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:40.513766 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:40.513662    2544 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 21:25:39 +0000 UTC" deadline="2028-05-01 00:29:23.776943102 +0000 UTC"
Sep 11 21:30:40.513766 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:40.513714    2544 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14330h58m43.263231579s"
Sep 11 21:30:40.530303 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:40.530270    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1d2ce1_6b6f_43cd_a939_0fd6ae99cf6e.slice/crio-0b9a9f587f49e17ad3c5d047c6423bb9b1269d6f5e203e86a3b827e0e4783816 WatchSource:0}: Error finding container 0b9a9f587f49e17ad3c5d047c6423bb9b1269d6f5e203e86a3b827e0e4783816: Status 404 returned error can't find the container with id 0b9a9f587f49e17ad3c5d047c6423bb9b1269d6f5e203e86a3b827e0e4783816
Sep 11 21:30:40.531933 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:40.531904    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcecafdee_703e_498c_91ad_573fa4e60d2d.slice/crio-928794a44435d6ab7fe8f3c8b777882d1adfd43aa02ac365d5d2c30d0660143d WatchSource:0}: Error finding container 928794a44435d6ab7fe8f3c8b777882d1adfd43aa02ac365d5d2c30d0660143d: Status 404 returned error can't find the container with id 928794a44435d6ab7fe8f3c8b777882d1adfd43aa02ac365d5d2c30d0660143d
Sep 11 21:30:40.536505 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:40.536476    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c49e47_3378_4c6c_984a_802481fbf22c.slice/crio-33cbd0f31989ad9f871206d06db39f367514c9c0ca55db005614a891217d8ea5 WatchSource:0}: Error finding container 33cbd0f31989ad9f871206d06db39f367514c9c0ca55db005614a891217d8ea5: Status 404 returned error can't find the container with id 33cbd0f31989ad9f871206d06db39f367514c9c0ca55db005614a891217d8ea5
Sep 11 21:30:40.537294 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:40.537201    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c045b9b_c992_4a35_ba9d_0769d9e22dae.slice/crio-3dcb0bdc4d88b64deec58551e7e964ed7b104d43ef5a8a5991c50459f4f8b4c5 WatchSource:0}: Error finding container 3dcb0bdc4d88b64deec58551e7e964ed7b104d43ef5a8a5991c50459f4f8b4c5: Status 404 returned error can't find the container with id 3dcb0bdc4d88b64deec58551e7e964ed7b104d43ef5a8a5991c50459f4f8b4c5
Sep 11 21:30:40.538882 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:40.538477    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba240ac_88f7_4d2c_80c2_4567bb23184b.slice/crio-718b46a15c9f67b567bed6c0464bf50af9ebde3e2525fd81072a6441e97560e7 WatchSource:0}: Error finding container 718b46a15c9f67b567bed6c0464bf50af9ebde3e2525fd81072a6441e97560e7: Status 404 returned error can't find the container with id 718b46a15c9f67b567bed6c0464bf50af9ebde3e2525fd81072a6441e97560e7
Sep 11 21:30:40.539338 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:40.539157    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec9b8d7_5c6f_4fcf_8dfa_b2e93544f8bd.slice/crio-c72004c81ca2dad2017bc2760e40d708b25e705e26adcba22a2d5769abb9e4c9 WatchSource:0}: Error finding container c72004c81ca2dad2017bc2760e40d708b25e705e26adcba22a2d5769abb9e4c9: Status 404 returned error can't find the container with id c72004c81ca2dad2017bc2760e40d708b25e705e26adcba22a2d5769abb9e4c9
Sep 11 21:30:40.540851 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:40.540825    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec47591_6ba3_441d_8541_ee68a0470a2b.slice/crio-e62fd600eac2f52cc518b4cc2e18931e7c0d51197e044286ca9b99db1de5e3ec WatchSource:0}: Error finding container e62fd600eac2f52cc518b4cc2e18931e7c0d51197e044286ca9b99db1de5e3ec: Status 404 returned error can't find the container with id e62fd600eac2f52cc518b4cc2e18931e7c0d51197e044286ca9b99db1de5e3ec
Sep 11 21:30:40.542336 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:40.542314    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44d4afa_8d65_4feb_83f9_7f6ed664511a.slice/crio-db20abc2eb019853006de288551d537589022561bb1b18cd8e17dad7b85f04fd WatchSource:0}: Error finding container db20abc2eb019853006de288551d537589022561bb1b18cd8e17dad7b85f04fd: Status 404 returned error can't find the container with id db20abc2eb019853006de288551d537589022561bb1b18cd8e17dad7b85f04fd
Sep 11 21:30:40.543705 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:30:40.543498    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728ca319_dbd4_4ade_8a93_10191019f4df.slice/crio-d229707b9655d0cac5690c115714cab7ce0e85d238db89c625d0b97724649a49 WatchSource:0}: Error finding container d229707b9655d0cac5690c115714cab7ce0e85d238db89c625d0b97724649a49: Status 404 returned error can't find the container with id d229707b9655d0cac5690c115714cab7ce0e85d238db89c625d0b97724649a49
Sep 11 21:30:40.586331 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:40.586305    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:30:40.586463 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:40.586399    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmjnx" podUID="6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b"
Sep 11 21:30:40.586463 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:40.586415    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:40.586559 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:40.586495    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjvb6" podUID="fef5eac9-cf6d-40b0-9bb2-374516793dc2"
Sep 11 21:30:40.598426 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:40.598251    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sh5wj" event={"ID":"728ca319-dbd4-4ade-8a93-10191019f4df","Type":"ContainerStarted","Data":"d229707b9655d0cac5690c115714cab7ce0e85d238db89c625d0b97724649a49"}
Sep 11 21:30:40.599472 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:40.599443    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qrcsr" event={"ID":"8ba240ac-88f7-4d2c-80c2-4567bb23184b","Type":"ContainerStarted","Data":"718b46a15c9f67b567bed6c0464bf50af9ebde3e2525fd81072a6441e97560e7"}
Sep 11 21:30:40.600730 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:40.600706    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7" event={"ID":"9c045b9b-c992-4a35-ba9d-0769d9e22dae","Type":"ContainerStarted","Data":"3dcb0bdc4d88b64deec58551e7e964ed7b104d43ef5a8a5991c50459f4f8b4c5"}
Sep 11 21:30:40.601849 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:40.601831    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6gf57" event={"ID":"07c49e47-3378-4c6c-984a-802481fbf22c","Type":"ContainerStarted","Data":"33cbd0f31989ad9f871206d06db39f367514c9c0ca55db005614a891217d8ea5"}
Sep 11 21:30:40.602937 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:40.602916    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lfxpn" event={"ID":"cecafdee-703e-498c-91ad-573fa4e60d2d","Type":"ContainerStarted","Data":"928794a44435d6ab7fe8f3c8b777882d1adfd43aa02ac365d5d2c30d0660143d"}
Sep 11 21:30:40.603949 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:40.603927    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z8wtk" event={"ID":"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e","Type":"ContainerStarted","Data":"0b9a9f587f49e17ad3c5d047c6423bb9b1269d6f5e203e86a3b827e0e4783816"}
Sep 11 21:30:40.605041 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:40.605012    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rljhw" event={"ID":"c44d4afa-8d65-4feb-83f9-7f6ed664511a","Type":"ContainerStarted","Data":"db20abc2eb019853006de288551d537589022561bb1b18cd8e17dad7b85f04fd"}
Sep 11 21:30:40.606159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:40.606141    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dkd6r" event={"ID":"fec47591-6ba3-441d-8541-ee68a0470a2b","Type":"ContainerStarted","Data":"e62fd600eac2f52cc518b4cc2e18931e7c0d51197e044286ca9b99db1de5e3ec"}
Sep 11 21:30:40.607171 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:40.607155    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj" event={"ID":"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd","Type":"ContainerStarted","Data":"c72004c81ca2dad2017bc2760e40d708b25e705e26adcba22a2d5769abb9e4c9"}
Sep 11 21:30:41.080237 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:41.079963    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs\") pod \"network-metrics-daemon-k94pz\" (UID: \"047e6301-db2b-4c57-86ad-20ca80ff019a\") " pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:41.080237 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:41.080045    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret\") pod \"global-pull-secret-syncer-fjvb6\" (UID: \"fef5eac9-cf6d-40b0-9bb2-374516793dc2\") " pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:41.080443 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:41.080288    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:41.080443 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:41.080354    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret podName:fef5eac9-cf6d-40b0-9bb2-374516793dc2 nodeName:}" failed. No retries permitted until 2026-09-11 21:30:43.080336004 +0000 UTC m=+6.049752821 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret") pod "global-pull-secret-syncer-fjvb6" (UID: "fef5eac9-cf6d-40b0-9bb2-374516793dc2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:41.080846 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:41.080750    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:41.080846 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:41.080826    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs podName:047e6301-db2b-4c57-86ad-20ca80ff019a nodeName:}" failed. No retries permitted until 2026-09-11 21:30:43.080809222 +0000 UTC m=+6.050226034 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs") pod "network-metrics-daemon-k94pz" (UID: "047e6301-db2b-4c57-86ad-20ca80ff019a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:41.180586 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:41.180545    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl6th\" (UniqueName: \"kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th\") pod \"network-check-target-wmjnx\" (UID: \"6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b\") " pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:30:41.180774 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:41.180715    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:30:41.180774 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:41.180734    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:30:41.180774 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:41.180748    2544 projected.go:196] Error preparing data for projected volume kube-api-access-zl6th for pod openshift-network-diagnostics/network-check-target-wmjnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:41.180935 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:41.180804    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th podName:6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b nodeName:}" failed. No retries permitted until 2026-09-11 21:30:43.180786569 +0000 UTC m=+6.150203373 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zl6th" (UniqueName: "kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th") pod "network-check-target-wmjnx" (UID: "6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:41.514429 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:41.514321    2544 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 21:25:39 +0000 UTC" deadline="2028-06-07 14:19:45.110097504 +0000 UTC"
Sep 11 21:30:41.514429 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:41.514364    2544 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15232h49m3.595738504s"
Sep 11 21:30:41.585813 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:41.585779    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:41.585989 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:41.585948    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k94pz" podUID="047e6301-db2b-4c57-86ad-20ca80ff019a"
Sep 11 21:30:41.635971 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:41.635932    2544 generic.go:356] "Generic (PLEG): container finished" podID="7e17897ec24162ac27d9b7d9ee876183" containerID="e728e7d7cc3c2c9d6eb4f6057dc3677b7f6e7ae960ab9ad71acd29e7508112ad" exitCode=0
Sep 11 21:30:41.636136 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:41.636030    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal" event={"ID":"7e17897ec24162ac27d9b7d9ee876183","Type":"ContainerDied","Data":"e728e7d7cc3c2c9d6eb4f6057dc3677b7f6e7ae960ab9ad71acd29e7508112ad"}
Sep 11 21:30:41.660917 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:41.660832    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-211.ec2.internal" event={"ID":"400563a539e706696660090857d9ffd0","Type":"ContainerStarted","Data":"ef73e8ada27155fca6000c38d3154f484307e4448d01b88dba294ef04ebc8edc"}
Sep 11 21:30:41.677522 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:41.677462    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-211.ec2.internal" podStartSLOduration=3.677439638 podStartE2EDuration="3.677439638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:41.676605033 +0000 UTC m=+4.646021855" watchObservedRunningTime="2026-09-11 21:30:41.677439638 +0000 UTC m=+4.646856459"
Sep 11 21:30:42.586786 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:42.586700    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:30:42.587384 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:42.586834    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmjnx" podUID="6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b"
Sep 11 21:30:42.587906 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:42.587873    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:42.588001 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:42.587986    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjvb6" podUID="fef5eac9-cf6d-40b0-9bb2-374516793dc2"
Sep 11 21:30:42.695782 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:42.695750    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal" event={"ID":"7e17897ec24162ac27d9b7d9ee876183","Type":"ContainerStarted","Data":"5c9752a42f904eaf232a277bf2b726b7da35c784d8a41a80f529e9ae4f0d4e6a"}
Sep 11 21:30:42.713334 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:42.713278    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-211.ec2.internal" podStartSLOduration=4.713249444 podStartE2EDuration="4.713249444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:42.712136094 +0000 UTC m=+5.681552906" watchObservedRunningTime="2026-09-11 21:30:42.713249444 +0000 UTC m=+5.682666262"
Sep 11 21:30:43.094998 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:43.094950    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs\") pod \"network-metrics-daemon-k94pz\" (UID: \"047e6301-db2b-4c57-86ad-20ca80ff019a\") " pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:43.095190 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:43.095040    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret\") pod \"global-pull-secret-syncer-fjvb6\" (UID: \"fef5eac9-cf6d-40b0-9bb2-374516793dc2\") " pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:43.095190 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:43.095178    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:43.095330 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:43.095259    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret podName:fef5eac9-cf6d-40b0-9bb2-374516793dc2 nodeName:}" failed. No retries permitted until 2026-09-11 21:30:47.095238626 +0000 UTC m=+10.064655440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret") pod "global-pull-secret-syncer-fjvb6" (UID: "fef5eac9-cf6d-40b0-9bb2-374516793dc2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:43.095897 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:43.095804    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:43.095897 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:43.095869    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs podName:047e6301-db2b-4c57-86ad-20ca80ff019a nodeName:}" failed. No retries permitted until 2026-09-11 21:30:47.095853119 +0000 UTC m=+10.065269919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs") pod "network-metrics-daemon-k94pz" (UID: "047e6301-db2b-4c57-86ad-20ca80ff019a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:43.196477 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:43.196426    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl6th\" (UniqueName: \"kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th\") pod \"network-check-target-wmjnx\" (UID: \"6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b\") " pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:30:43.196660 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:43.196599    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:30:43.196660 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:43.196619    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:30:43.196660 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:43.196631    2544 projected.go:196] Error preparing data for projected volume kube-api-access-zl6th for pod openshift-network-diagnostics/network-check-target-wmjnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:43.196809 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:43.196690    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th podName:6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b nodeName:}" failed. No retries permitted until 2026-09-11 21:30:47.196671006 +0000 UTC m=+10.166087808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zl6th" (UniqueName: "kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th") pod "network-check-target-wmjnx" (UID: "6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:43.587412 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:43.586811    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:43.587412 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:43.586957    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k94pz" podUID="047e6301-db2b-4c57-86ad-20ca80ff019a"
Sep 11 21:30:44.586133 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:44.586095    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:30:44.586347 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:44.586139    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:44.586347 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:44.586262    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmjnx" podUID="6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b"
Sep 11 21:30:44.586471 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:44.586385    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjvb6" podUID="fef5eac9-cf6d-40b0-9bb2-374516793dc2"
Sep 11 21:30:45.586674 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:45.586635    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:45.587163 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:45.586808    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k94pz" podUID="047e6301-db2b-4c57-86ad-20ca80ff019a"
Sep 11 21:30:46.586602 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:46.586566    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:30:46.586798 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:46.586572    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:46.586798 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:46.586722    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmjnx" podUID="6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b"
Sep 11 21:30:46.586798 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:46.586780    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjvb6" podUID="fef5eac9-cf6d-40b0-9bb2-374516793dc2"
Sep 11 21:30:47.132567 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:47.132525    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs\") pod \"network-metrics-daemon-k94pz\" (UID: \"047e6301-db2b-4c57-86ad-20ca80ff019a\") " pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:47.132752 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:47.132615    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret\") pod \"global-pull-secret-syncer-fjvb6\" (UID: \"fef5eac9-cf6d-40b0-9bb2-374516793dc2\") " pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:47.132819 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:47.132756    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:47.132868 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:47.132822    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret podName:fef5eac9-cf6d-40b0-9bb2-374516793dc2 nodeName:}" failed. No retries permitted until 2026-09-11 21:30:55.132803011 +0000 UTC m=+18.102219826 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret") pod "global-pull-secret-syncer-fjvb6" (UID: "fef5eac9-cf6d-40b0-9bb2-374516793dc2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:47.132926 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:47.132904    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:47.132978 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:47.132968    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs podName:047e6301-db2b-4c57-86ad-20ca80ff019a nodeName:}" failed. No retries permitted until 2026-09-11 21:30:55.13295165 +0000 UTC m=+18.102368461 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs") pod "network-metrics-daemon-k94pz" (UID: "047e6301-db2b-4c57-86ad-20ca80ff019a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:47.233870 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:47.233822    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl6th\" (UniqueName: \"kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th\") pod \"network-check-target-wmjnx\" (UID: \"6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b\") " pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:30:47.234111 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:47.233990    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:30:47.234111 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:47.234009    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:30:47.234111 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:47.234022    2544 projected.go:196] Error preparing data for projected volume kube-api-access-zl6th for pod openshift-network-diagnostics/network-check-target-wmjnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:47.234299 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:47.234117    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th podName:6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b nodeName:}" failed. No retries permitted until 2026-09-11 21:30:55.234098274 +0000 UTC m=+18.203515075 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zl6th" (UniqueName: "kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th") pod "network-check-target-wmjnx" (UID: "6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:47.590049 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:47.589886    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:47.590049 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:47.590017    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k94pz" podUID="047e6301-db2b-4c57-86ad-20ca80ff019a"
Sep 11 21:30:48.585988 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:48.585953    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:30:48.586164 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:48.585965    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:48.586164 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:48.586065    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmjnx" podUID="6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b"
Sep 11 21:30:48.586288 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:48.586167    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjvb6" podUID="fef5eac9-cf6d-40b0-9bb2-374516793dc2"
Sep 11 21:30:49.586819 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:49.586783    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:49.587392 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:49.586958    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k94pz" podUID="047e6301-db2b-4c57-86ad-20ca80ff019a"
Sep 11 21:30:50.586280 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:50.586244    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:30:50.586472 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:50.586244    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:50.586472 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:50.586361    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmjnx" podUID="6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b"
Sep 11 21:30:50.586472 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:50.586456    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjvb6" podUID="fef5eac9-cf6d-40b0-9bb2-374516793dc2"
Sep 11 21:30:51.586280 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:51.586239    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:51.586747 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:51.586396    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k94pz" podUID="047e6301-db2b-4c57-86ad-20ca80ff019a"
Sep 11 21:30:52.586730 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:52.586690    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:52.587148 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:52.586690    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:30:52.587148 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:52.586813    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjvb6" podUID="fef5eac9-cf6d-40b0-9bb2-374516793dc2"
Sep 11 21:30:52.587148 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:52.586903    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmjnx" podUID="6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b"
Sep 11 21:30:53.586077 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:53.585957    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:53.586285 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:53.586115    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k94pz" podUID="047e6301-db2b-4c57-86ad-20ca80ff019a"
Sep 11 21:30:54.586359 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:54.586322    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:30:54.586917 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:54.586322    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:54.586917 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:54.586446    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmjnx" podUID="6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b"
Sep 11 21:30:54.586917 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:54.586507    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjvb6" podUID="fef5eac9-cf6d-40b0-9bb2-374516793dc2"
Sep 11 21:30:55.196551 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:55.196507    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs\") pod \"network-metrics-daemon-k94pz\" (UID: \"047e6301-db2b-4c57-86ad-20ca80ff019a\") " pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:55.196739 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:55.196673    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret\") pod \"global-pull-secret-syncer-fjvb6\" (UID: \"fef5eac9-cf6d-40b0-9bb2-374516793dc2\") " pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:55.196739 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:55.196681    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:55.196847 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:55.196758    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs podName:047e6301-db2b-4c57-86ad-20ca80ff019a nodeName:}" failed. No retries permitted until 2026-09-11 21:31:11.196736565 +0000 UTC m=+34.166153364 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs") pod "network-metrics-daemon-k94pz" (UID: "047e6301-db2b-4c57-86ad-20ca80ff019a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:55.196847 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:55.196765    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:55.196847 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:55.196811    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret podName:fef5eac9-cf6d-40b0-9bb2-374516793dc2 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:11.196796067 +0000 UTC m=+34.166212865 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret") pod "global-pull-secret-syncer-fjvb6" (UID: "fef5eac9-cf6d-40b0-9bb2-374516793dc2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:55.297239 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:55.297189    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl6th\" (UniqueName: \"kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th\") pod \"network-check-target-wmjnx\" (UID: \"6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b\") " pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:30:55.297410 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:55.297389    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:30:55.297468 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:55.297417    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:30:55.297468 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:55.297432    2544 projected.go:196] Error preparing data for projected volume kube-api-access-zl6th for pod openshift-network-diagnostics/network-check-target-wmjnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:55.297566 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:55.297513    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th podName:6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b nodeName:}" failed. No retries permitted until 2026-09-11 21:31:11.297498341 +0000 UTC m=+34.266915139 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zl6th" (UniqueName: "kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th") pod "network-check-target-wmjnx" (UID: "6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:55.586030 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:55.585950    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:55.586194 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:55.586071    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k94pz" podUID="047e6301-db2b-4c57-86ad-20ca80ff019a"
Sep 11 21:30:56.586448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:56.586415    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:30:56.586448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:56.586457    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:56.586928 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:56.586542    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmjnx" podUID="6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b"
Sep 11 21:30:56.586928 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:56.586695    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjvb6" podUID="fef5eac9-cf6d-40b0-9bb2-374516793dc2"
Sep 11 21:30:57.586851 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:57.586811    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:57.587323 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:57.586963    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k94pz" podUID="047e6301-db2b-4c57-86ad-20ca80ff019a"
Sep 11 21:30:58.586413 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:58.585830    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:30:58.586413 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:58.585967    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjvb6" podUID="fef5eac9-cf6d-40b0-9bb2-374516793dc2"
Sep 11 21:30:58.586413 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:58.586046    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:30:58.586413 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:58.586119    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmjnx" podUID="6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b"
Sep 11 21:30:58.729513 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:58.729256    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sh5wj" event={"ID":"728ca319-dbd4-4ade-8a93-10191019f4df","Type":"ContainerStarted","Data":"4ed70d0fa6a100108e5950faabf91139786f25b53c648d187fd4efa366a044c4"}
Sep 11 21:30:58.731123 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:58.731092    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7" event={"ID":"9c045b9b-c992-4a35-ba9d-0769d9e22dae","Type":"ContainerStarted","Data":"6363e2b7f8dd7ed379d2e40feb6fe9a76485e22e70d8762919857238ae7f21e6"}
Sep 11 21:30:58.732613 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:58.732584    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6gf57" event={"ID":"07c49e47-3378-4c6c-984a-802481fbf22c","Type":"ContainerStarted","Data":"dc621e9945fee6c218ca9d42ace5c1238d0e493f3591034f797418ff42cdde5c"}
Sep 11 21:30:58.734244 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:58.734203    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lfxpn" event={"ID":"cecafdee-703e-498c-91ad-573fa4e60d2d","Type":"ContainerStarted","Data":"bf437de9a5d03aee86b6eb086e4c14ace2bc6203d30d9fe4dbd01f6607bf3ae5"}
Sep 11 21:30:58.735776 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:58.735747    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z8wtk" event={"ID":"5e1d2ce1-6b6f-43cd-a939-0fd6ae99cf6e","Type":"ContainerStarted","Data":"d246d49ee9941e6c6fed9067363d827dc0fe631c23528b6ca77cdd935cf21cf6"}
Sep 11 21:30:58.737299 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:58.737278    2544 generic.go:356] "Generic (PLEG): container finished" podID="c44d4afa-8d65-4feb-83f9-7f6ed664511a" containerID="756985748afc0805476cd8e89cf72793f095f0bbc52a67edb6751d30e20d02f4" exitCode=0
Sep 11 21:30:58.737398 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:58.737340    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rljhw" event={"ID":"c44d4afa-8d65-4feb-83f9-7f6ed664511a","Type":"ContainerDied","Data":"756985748afc0805476cd8e89cf72793f095f0bbc52a67edb6751d30e20d02f4"}
Sep 11 21:30:58.739076 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:58.739030    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dkd6r" event={"ID":"fec47591-6ba3-441d-8541-ee68a0470a2b","Type":"ContainerStarted","Data":"5875ba75227c5c8532082ade61d2c7d30f15e3cff6cd618a7d2d0a4890cbd50b"}
Sep 11 21:30:58.741168 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:58.741143    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj" event={"ID":"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd","Type":"ContainerStarted","Data":"60743dc43075697492cdf79f083bde075695ab9d67eb1130dc4501e44308d6c7"}
Sep 11 21:30:58.741265 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:58.741241    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj" event={"ID":"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd","Type":"ContainerStarted","Data":"5de383af62ca1dd27cbc81c63354cca26c22d0ae78396b7baa94d5d071d808e5"}
Sep 11 21:30:58.744740 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:58.744699    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sh5wj" podStartSLOduration=3.882029805 podStartE2EDuration="21.74468896s" totalImagesPullingTime="17.862659155s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:58.744566039 +0000 UTC m=+21.713982872" watchObservedRunningTime="2026-09-11 21:30:58.74468896 +0000 UTC m=+21.714105780"
Sep 11 21:30:58.759296 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:58.759251    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6gf57" podStartSLOduration=3.8894895 podStartE2EDuration="21.75923394s" totalImagesPullingTime="17.86974444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:58.758844055 +0000 UTC m=+21.728260877" watchObservedRunningTime="2026-09-11 21:30:58.75923394 +0000 UTC m=+21.728650758"
Sep 11 21:30:58.774638 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:58.774596    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dkd6r" podStartSLOduration=11.421533484 podStartE2EDuration="21.774579473s" totalImagesPullingTime="10.353045989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:58.774392491 +0000 UTC m=+21.743809312" watchObservedRunningTime="2026-09-11 21:30:58.774579473 +0000 UTC m=+21.743996293"
Sep 11 21:30:58.798170 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:58.796733    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lfxpn" podStartSLOduration=3.7639543189999998 podStartE2EDuration="21.796709517s" totalImagesPullingTime="18.032755198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:58.795572304 +0000 UTC m=+21.764989125" watchObservedRunningTime="2026-09-11 21:30:58.796709517 +0000 UTC m=+21.766126338"
Sep 11 21:30:58.836422 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:58.836378    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z8wtk" podStartSLOduration=3.958851364 podStartE2EDuration="21.83636153s" totalImagesPullingTime="17.877510166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:58.836147748 +0000 UTC m=+21.805564570" watchObservedRunningTime="2026-09-11 21:30:58.83636153 +0000 UTC m=+21.805778376"
Sep 11 21:30:59.586086 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:59.586047    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:30:59.586284 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:30:59.586205    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k94pz" podUID="047e6301-db2b-4c57-86ad-20ca80ff019a"
Sep 11 21:30:59.747083 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:59.747037    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj" event={"ID":"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd","Type":"ContainerStarted","Data":"799027be118e3e998f3d94c0ca4d2036c2b5b03966b075a78591a67e9f7fd868"}
Sep 11 21:30:59.747873 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:59.747129    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj" event={"ID":"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd","Type":"ContainerStarted","Data":"5ddf58e1cae4ab15095df3d4732f86c3553c333b9d0d025d1ed46059a4c00246"}
Sep 11 21:30:59.747873 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:59.747174    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj" event={"ID":"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd","Type":"ContainerStarted","Data":"3c0c9c28712dcdfea7ce857cf8e8041582623d7d3f6d560331ce79f40ca28d3a"}
Sep 11 21:30:59.747873 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:59.747246    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj" event={"ID":"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd","Type":"ContainerStarted","Data":"903d9a22a6238221c5c5f1b70d410c45c2c7467842edf2a61a733667f8286ccc"}
Sep 11 21:30:59.962561 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:30:59.962489    2544 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 21:31:00.523361 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:00.523241    2544 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T21:30:59.962509495Z","UUID":"491c3fe0-de18-4cc0-97df-167ae485ec7f","Handler":null,"Name":"","Endpoint":""}
Sep 11 21:31:00.525070 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:00.525048    2544 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 21:31:00.525180 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:00.525084    2544 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 21:31:00.586250 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:00.586194    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:31:00.586545 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:00.586265    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:31:00.586650 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:00.586580    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmjnx" podUID="6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b"
Sep 11 21:31:00.586650 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:00.586619    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjvb6" podUID="fef5eac9-cf6d-40b0-9bb2-374516793dc2"
Sep 11 21:31:00.752324 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:00.752286    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qrcsr" event={"ID":"8ba240ac-88f7-4d2c-80c2-4567bb23184b","Type":"ContainerStarted","Data":"0eef0435de59b7cceba2d1b3baf0aba8a172a578b9a187d850df0de4db08b896"}
Sep 11 21:31:00.757748 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:00.757717    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7" event={"ID":"9c045b9b-c992-4a35-ba9d-0769d9e22dae","Type":"ContainerStarted","Data":"3a60b3cebd0586657a929b3539bc84f70cd7b8233a343be6bdbcb3e3ee7f57e9"}
Sep 11 21:31:00.767473 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:00.767428    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qrcsr" podStartSLOduration=5.899565669 podStartE2EDuration="23.767412084s" totalImagesPullingTime="17.867846415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:00.767355191 +0000 UTC m=+23.736772011" watchObservedRunningTime="2026-09-11 21:31:00.767412084 +0000 UTC m=+23.736828904"
Sep 11 21:31:01.586351 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:01.586313    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:31:01.586539 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:01.586458    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k94pz" podUID="047e6301-db2b-4c57-86ad-20ca80ff019a"
Sep 11 21:31:01.762514 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:01.762473    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7" event={"ID":"9c045b9b-c992-4a35-ba9d-0769d9e22dae","Type":"ContainerStarted","Data":"ee0a9feb85efd0219dd58ed96e5f9912e666962e2ae336883041c184a62d417f"}
Sep 11 21:31:01.766494 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:01.766463    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj" event={"ID":"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd","Type":"ContainerStarted","Data":"d016899c4f83f8da672eb5a5dfa5aaa81028ac87e476350ac87e7a99b368ae78"}
Sep 11 21:31:01.782171 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:01.782108    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng8h7" podStartSLOduration=4.710302076 podStartE2EDuration="24.782088385s" totalImagesPullingTime="20.071786309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:01.781526324 +0000 UTC m=+24.750943147" watchObservedRunningTime="2026-09-11 21:31:01.782088385 +0000 UTC m=+24.751505252"
Sep 11 21:31:02.204871 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:02.204828    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dkd6r"
Sep 11 21:31:02.543189 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:02.542989    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dkd6r"
Sep 11 21:31:02.543633 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:02.543615    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dkd6r"
Sep 11 21:31:02.586147 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:02.586115    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:31:02.586147 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:02.586143    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:31:02.586369 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:02.586241    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmjnx" podUID="6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b"
Sep 11 21:31:02.586369 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:02.586356    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjvb6" podUID="fef5eac9-cf6d-40b0-9bb2-374516793dc2"
Sep 11 21:31:02.769996 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:02.769968    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dkd6r"
Sep 11 21:31:03.586549 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:03.586512    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:31:03.586739 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:03.586647    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k94pz" podUID="047e6301-db2b-4c57-86ad-20ca80ff019a"
Sep 11 21:31:03.773907 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:03.773872    2544 generic.go:356] "Generic (PLEG): container finished" podID="c44d4afa-8d65-4feb-83f9-7f6ed664511a" containerID="f3d878d0cd67fd9a9ea351c31acab99c125a14d9aace8bc1298c7276a6f8d91e" exitCode=0
Sep 11 21:31:03.774404 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:03.773958    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rljhw" event={"ID":"c44d4afa-8d65-4feb-83f9-7f6ed664511a","Type":"ContainerDied","Data":"f3d878d0cd67fd9a9ea351c31acab99c125a14d9aace8bc1298c7276a6f8d91e"}
Sep 11 21:31:03.777996 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:03.777961    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj" event={"ID":"5ec9b8d7-5c6f-4fcf-8dfa-b2e93544f8bd","Type":"ContainerStarted","Data":"e102b56e5a0ed5b46677873158191092852ba9eabb064126dffae04b1c3921bd"}
Sep 11 21:31:03.831737 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:03.831681    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj" podStartSLOduration=8.943346398 podStartE2EDuration="26.831665228s" totalImagesPullingTime="17.88831883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:03.830480918 +0000 UTC m=+26.799897738" watchObservedRunningTime="2026-09-11 21:31:03.831665228 +0000 UTC m=+26.801082047"
Sep 11 21:31:04.586574 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:04.586376    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:31:04.586848 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:04.586376    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:31:04.586848 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:04.586642    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjvb6" podUID="fef5eac9-cf6d-40b0-9bb2-374516793dc2"
Sep 11 21:31:04.586848 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:04.586736    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmjnx" podUID="6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b"
Sep 11 21:31:04.780048 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:04.780021    2544 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 21:31:04.780625 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:04.780599    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:31:04.780731 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:04.780692    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:31:04.796040 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:04.796015    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:31:04.799821 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:04.799800    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:31:05.211044 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:05.211003    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:31:05.311853 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:05.311815    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wmjnx"]
Sep 11 21:31:05.312009 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:05.311938    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:31:05.312058 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:05.312018    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmjnx" podUID="6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b"
Sep 11 21:31:05.316027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:05.315944    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fjvb6"]
Sep 11 21:31:05.316203 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:05.316105    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:31:05.316289 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:05.316233    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjvb6" podUID="fef5eac9-cf6d-40b0-9bb2-374516793dc2"
Sep 11 21:31:05.316623 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:05.316601    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k94pz"]
Sep 11 21:31:05.316735 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:05.316716    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:31:05.316854 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:05.316817    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k94pz" podUID="047e6301-db2b-4c57-86ad-20ca80ff019a"
Sep 11 21:31:05.784603 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:05.784572    2544 generic.go:356] "Generic (PLEG): container finished" podID="c44d4afa-8d65-4feb-83f9-7f6ed664511a" containerID="597d28183e533facbf59f4cbdc3a89e0b2ce608e4b3bd6d152e24441ccd00b00" exitCode=0
Sep 11 21:31:05.785384 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:05.784655    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rljhw" event={"ID":"c44d4afa-8d65-4feb-83f9-7f6ed664511a","Type":"ContainerDied","Data":"597d28183e533facbf59f4cbdc3a89e0b2ce608e4b3bd6d152e24441ccd00b00"}
Sep 11 21:31:06.586424 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:06.586402    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:31:06.586504 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:06.586431    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:31:06.586540 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:06.586516    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k94pz" podUID="047e6301-db2b-4c57-86ad-20ca80ff019a"
Sep 11 21:31:06.586641 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:06.586623    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjvb6" podUID="fef5eac9-cf6d-40b0-9bb2-374516793dc2"
Sep 11 21:31:06.789828 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:06.789792    2544 generic.go:356] "Generic (PLEG): container finished" podID="c44d4afa-8d65-4feb-83f9-7f6ed664511a" containerID="5451201ee26fca1777fa3e37311cd80e0a55c72b6e0897f1c850040a7d695778" exitCode=0
Sep 11 21:31:06.790260 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:06.789879    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rljhw" event={"ID":"c44d4afa-8d65-4feb-83f9-7f6ed664511a","Type":"ContainerDied","Data":"5451201ee26fca1777fa3e37311cd80e0a55c72b6e0897f1c850040a7d695778"}
Sep 11 21:31:07.587526 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:07.587201    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:31:07.587671 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:07.587576    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmjnx" podUID="6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b"
Sep 11 21:31:08.586047 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:08.586008    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:31:08.586546 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:08.586008    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:31:08.586546 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:08.586158    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k94pz" podUID="047e6301-db2b-4c57-86ad-20ca80ff019a"
Sep 11 21:31:08.586546 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:08.586266    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjvb6" podUID="fef5eac9-cf6d-40b0-9bb2-374516793dc2"
Sep 11 21:31:09.586512 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:09.586476    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:31:09.587001 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:09.586614    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmjnx" podUID="6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b"
Sep 11 21:31:10.586592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:10.586549    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:31:10.587091 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:10.586695    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k94pz" podUID="047e6301-db2b-4c57-86ad-20ca80ff019a"
Sep 11 21:31:10.587091 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:10.586755    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:31:10.587091 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:10.586885    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjvb6" podUID="fef5eac9-cf6d-40b0-9bb2-374516793dc2"
Sep 11 21:31:11.217803 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.217760    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret\") pod \"global-pull-secret-syncer-fjvb6\" (UID: \"fef5eac9-cf6d-40b0-9bb2-374516793dc2\") " pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:31:11.217985 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.217830    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs\") pod \"network-metrics-daemon-k94pz\" (UID: \"047e6301-db2b-4c57-86ad-20ca80ff019a\") " pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:31:11.217985 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:11.217925    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:31:11.217985 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:11.217937    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:31:11.218123 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:11.217995    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret podName:fef5eac9-cf6d-40b0-9bb2-374516793dc2 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:43.217979684 +0000 UTC m=+66.187396487 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret") pod "global-pull-secret-syncer-fjvb6" (UID: "fef5eac9-cf6d-40b0-9bb2-374516793dc2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:31:11.218123 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:11.218014    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs podName:047e6301-db2b-4c57-86ad-20ca80ff019a nodeName:}" failed. No retries permitted until 2026-09-11 21:31:43.218006533 +0000 UTC m=+66.187423331 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs") pod "network-metrics-daemon-k94pz" (UID: "047e6301-db2b-4c57-86ad-20ca80ff019a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:31:11.258755 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.258732    2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-211.ec2.internal" event="NodeReady"
Sep 11 21:31:11.258915 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.258868    2544 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 21:31:11.316481 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.316446    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pnzdj"]
Sep 11 21:31:11.318952 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.318927    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl6th\" (UniqueName: \"kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th\") pod \"network-check-target-wmjnx\" (UID: \"6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b\") " pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:31:11.319116 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:11.319099    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:31:11.319163 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:11.319120    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:31:11.319163 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:11.319130    2544 projected.go:196] Error preparing data for projected volume kube-api-access-zl6th for pod openshift-network-diagnostics/network-check-target-wmjnx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:31:11.319238 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:11.319183    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th podName:6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b nodeName:}" failed. No retries permitted until 2026-09-11 21:31:43.31916637 +0000 UTC m=+66.288583168 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zl6th" (UniqueName: "kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th") pod "network-check-target-wmjnx" (UID: "6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:31:11.332689 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.332654    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dprjf"]
Sep 11 21:31:11.332857 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.332764    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:31:11.335579 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.335545    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 21:31:11.335718 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.335601    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-59g8b\""
Sep 11 21:31:11.335908 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.335888    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 21:31:11.346698 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.346670    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pnzdj"]
Sep 11 21:31:11.346836 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.346707    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dprjf"]
Sep 11 21:31:11.346836 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.346794    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dprjf"
Sep 11 21:31:11.349492 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.349474    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 21:31:11.349895 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.349877    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vcb9z\""
Sep 11 21:31:11.350130 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.350112    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 21:31:11.350189 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.350143    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 21:31:11.419410 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.419366    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqspr\" (UniqueName: \"kubernetes.io/projected/ae83570c-f2ee-4671-818e-f68d8c4cd4d0-kube-api-access-lqspr\") pod \"ingress-canary-dprjf\" (UID: \"ae83570c-f2ee-4671-818e-f68d8c4cd4d0\") " pod="openshift-ingress-canary/ingress-canary-dprjf"
Sep 11 21:31:11.419410 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.419413    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/77e62236-28b9-4261-85cb-b2aba118874b-config-volume\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:31:11.419622 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.419503    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e62236-28b9-4261-85cb-b2aba118874b-tmp-dir\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:31:11.419622 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.419553    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqd7k\" (UniqueName: \"kubernetes.io/projected/77e62236-28b9-4261-85cb-b2aba118874b-kube-api-access-vqd7k\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:31:11.419622 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.419575    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae83570c-f2ee-4671-818e-f68d8c4cd4d0-cert\") pod \"ingress-canary-dprjf\" (UID: \"ae83570c-f2ee-4671-818e-f68d8c4cd4d0\") " pod="openshift-ingress-canary/ingress-canary-dprjf"
Sep 11 21:31:11.419622 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.419593    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:31:11.520833 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.520757    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e62236-28b9-4261-85cb-b2aba118874b-tmp-dir\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:31:11.520833 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.520798    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqd7k\" (UniqueName: \"kubernetes.io/projected/77e62236-28b9-4261-85cb-b2aba118874b-kube-api-access-vqd7k\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:31:11.520833 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.520819    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae83570c-f2ee-4671-818e-f68d8c4cd4d0-cert\") pod \"ingress-canary-dprjf\" (UID: \"ae83570c-f2ee-4671-818e-f68d8c4cd4d0\") " pod="openshift-ingress-canary/ingress-canary-dprjf"
Sep 11 21:31:11.520833 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.520835    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:31:11.521048 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.520894    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqspr\" (UniqueName: \"kubernetes.io/projected/ae83570c-f2ee-4671-818e-f68d8c4cd4d0-kube-api-access-lqspr\") pod \"ingress-canary-dprjf\" (UID: \"ae83570c-f2ee-4671-818e-f68d8c4cd4d0\") " pod="openshift-ingress-canary/ingress-canary-dprjf"
Sep 11 21:31:11.521048 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.520916    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/77e62236-28b9-4261-85cb-b2aba118874b-config-volume\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:31:11.521144 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:11.521109    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:31:11.521192 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.521173    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e62236-28b9-4261-85cb-b2aba118874b-tmp-dir\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:31:11.521301 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:11.521192    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls podName:77e62236-28b9-4261-85cb-b2aba118874b nodeName:}" failed. No retries permitted until 2026-09-11 21:31:12.021170727 +0000 UTC m=+34.990587549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls") pod "dns-default-pnzdj" (UID: "77e62236-28b9-4261-85cb-b2aba118874b") : secret "dns-default-metrics-tls" not found
Sep 11 21:31:11.521474 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.521456    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/77e62236-28b9-4261-85cb-b2aba118874b-config-volume\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:31:11.525063 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.525037    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae83570c-f2ee-4671-818e-f68d8c4cd4d0-cert\") pod \"ingress-canary-dprjf\" (UID: \"ae83570c-f2ee-4671-818e-f68d8c4cd4d0\") " pod="openshift-ingress-canary/ingress-canary-dprjf"
Sep 11 21:31:11.529960 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.529938    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqspr\" (UniqueName: \"kubernetes.io/projected/ae83570c-f2ee-4671-818e-f68d8c4cd4d0-kube-api-access-lqspr\") pod \"ingress-canary-dprjf\" (UID: \"ae83570c-f2ee-4671-818e-f68d8c4cd4d0\") " pod="openshift-ingress-canary/ingress-canary-dprjf"
Sep 11 21:31:11.530151 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.530131    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqd7k\" (UniqueName: \"kubernetes.io/projected/77e62236-28b9-4261-85cb-b2aba118874b-kube-api-access-vqd7k\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:31:11.586476 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.586441    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:31:11.589191 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.589164    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 21:31:11.589191 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.589181    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wxgvf\""
Sep 11 21:31:11.589630 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.589291    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 21:31:11.654808 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.654773    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dprjf"
Sep 11 21:31:11.807755 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.807442    2544 generic.go:356] "Generic (PLEG): container finished" podID="c44d4afa-8d65-4feb-83f9-7f6ed664511a" containerID="a2174ec911be070fdefc0cca9ea658bf25fa6ac138988f5c29ec84c0d4a9bea7" exitCode=0
Sep 11 21:31:11.807755 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.807520    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rljhw" event={"ID":"c44d4afa-8d65-4feb-83f9-7f6ed664511a","Type":"ContainerDied","Data":"a2174ec911be070fdefc0cca9ea658bf25fa6ac138988f5c29ec84c0d4a9bea7"}
Sep 11 21:31:11.978127 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:11.978095    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dprjf"]
Sep 11 21:31:12.025439 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:12.025400    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:31:12.025646 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:12.025560    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:31:12.025646 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:12.025632    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls podName:77e62236-28b9-4261-85cb-b2aba118874b nodeName:}" failed. No retries permitted until 2026-09-11 21:31:13.025610844 +0000 UTC m=+35.995027660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls") pod "dns-default-pnzdj" (UID: "77e62236-28b9-4261-85cb-b2aba118874b") : secret "dns-default-metrics-tls" not found
Sep 11 21:31:12.032939 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:31:12.032914    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae83570c_f2ee_4671_818e_f68d8c4cd4d0.slice/crio-b5079638a522b95d29e2ad432712a4f7a776779dc7b7cb24eb77f1d159e09d88 WatchSource:0}: Error finding container b5079638a522b95d29e2ad432712a4f7a776779dc7b7cb24eb77f1d159e09d88: Status 404 returned error can't find the container with id b5079638a522b95d29e2ad432712a4f7a776779dc7b7cb24eb77f1d159e09d88
Sep 11 21:31:12.585897 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:12.585858    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:31:12.586089 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:12.585858    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:31:12.590405 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:12.590380    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 21:31:12.590974 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:12.590433    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 21:31:12.590974 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:12.590765    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r57v4\""
Sep 11 21:31:12.813414 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:12.813379    2544 generic.go:356] "Generic (PLEG): container finished" podID="c44d4afa-8d65-4feb-83f9-7f6ed664511a" containerID="b308a3be70104fa6fd27382dcf15ce2b09d334462a977d17f319c330d2d6ab49" exitCode=0
Sep 11 21:31:12.813584 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:12.813462    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rljhw" event={"ID":"c44d4afa-8d65-4feb-83f9-7f6ed664511a","Type":"ContainerDied","Data":"b308a3be70104fa6fd27382dcf15ce2b09d334462a977d17f319c330d2d6ab49"}
Sep 11 21:31:12.814813 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:12.814788    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dprjf" event={"ID":"ae83570c-f2ee-4671-818e-f68d8c4cd4d0","Type":"ContainerStarted","Data":"b5079638a522b95d29e2ad432712a4f7a776779dc7b7cb24eb77f1d159e09d88"}
Sep 11 21:31:13.033280 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:13.033238    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:31:13.033466 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:13.033347    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:31:13.033466 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:13.033419    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls podName:77e62236-28b9-4261-85cb-b2aba118874b nodeName:}" failed. No retries permitted until 2026-09-11 21:31:15.033397105 +0000 UTC m=+38.002813921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls") pod "dns-default-pnzdj" (UID: "77e62236-28b9-4261-85cb-b2aba118874b") : secret "dns-default-metrics-tls" not found
Sep 11 21:31:13.821567 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:13.821531    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rljhw" event={"ID":"c44d4afa-8d65-4feb-83f9-7f6ed664511a","Type":"ContainerStarted","Data":"2869c867b03f8d624a31eb81eb1d24f1151b8dc484d8903a6222ba07343447bf"}
Sep 11 21:31:13.845350 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:13.845262    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rljhw" podStartSLOduration=8.483180228 podStartE2EDuration="36.845240548s" totalImagesPullingTime="28.36206032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:13.843735901 +0000 UTC m=+36.813152721" watchObservedRunningTime="2026-09-11 21:31:13.845240548 +0000 UTC m=+36.814657370"
Sep 11 21:31:14.826074 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:14.825875    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dprjf" event={"ID":"ae83570c-f2ee-4671-818e-f68d8c4cd4d0","Type":"ContainerStarted","Data":"7f22e0e6f549f7b764d0112b8cb6b16502c925639ff02f7c2b6c9b160e567989"}
Sep 11 21:31:14.842302 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:14.842247    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dprjf" podStartSLOduration=1.5558060660000002 podStartE2EDuration="3.842229438s" totalImagesPullingTime="2.286423372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:14.84201281 +0000 UTC m=+37.811429636" watchObservedRunningTime="2026-09-11 21:31:14.842229438 +0000 UTC m=+37.811646249"
Sep 11 21:31:15.048868 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:15.048832    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:31:15.049032 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:15.048975    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:31:15.049076 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:15.049036    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls podName:77e62236-28b9-4261-85cb-b2aba118874b nodeName:}" failed. No retries permitted until 2026-09-11 21:31:19.049018176 +0000 UTC m=+42.018434997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls") pod "dns-default-pnzdj" (UID: "77e62236-28b9-4261-85cb-b2aba118874b") : secret "dns-default-metrics-tls" not found
Sep 11 21:31:19.078349 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:19.078309    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:31:19.078755 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:19.078480    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:31:19.078755 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:19.078558    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls podName:77e62236-28b9-4261-85cb-b2aba118874b nodeName:}" failed. No retries permitted until 2026-09-11 21:31:27.078543633 +0000 UTC m=+50.047960445 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls") pod "dns-default-pnzdj" (UID: "77e62236-28b9-4261-85cb-b2aba118874b") : secret "dns-default-metrics-tls" not found
Sep 11 21:31:27.134234 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:27.134172    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:31:27.134655 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:27.134335    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:31:27.134655 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:27.134411    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls podName:77e62236-28b9-4261-85cb-b2aba118874b nodeName:}" failed. No retries permitted until 2026-09-11 21:31:43.134395584 +0000 UTC m=+66.103812402 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls") pod "dns-default-pnzdj" (UID: "77e62236-28b9-4261-85cb-b2aba118874b") : secret "dns-default-metrics-tls" not found
Sep 11 21:31:36.800540 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:36.800505    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5d2lj"
Sep 11 21:31:36.982744 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:36.982712    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"]
Sep 11 21:31:36.985803 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:36.985781    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"
Sep 11 21:31:36.991198 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:36.991150    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 11 21:31:36.991462 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:36.991443    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 21:31:36.992571 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:36.992549    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 11 21:31:36.992681 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:36.992618    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 21:31:36.992744 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:36.992674    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9242g\""
Sep 11 21:31:37.000193 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.000160    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"]
Sep 11 21:31:37.090162 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.090083    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx"]
Sep 11 21:31:37.091998 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.091983    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx"
Sep 11 21:31:37.095069 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.095041    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 11 21:31:37.095211 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.095185    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8lxhs\""
Sep 11 21:31:37.095468 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.095453    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 11 21:31:37.098610 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.098589    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d08eb046-1dfa-43cf-a851-0b722e18ba28-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-4pmkk\" (UID: \"d08eb046-1dfa-43cf-a851-0b722e18ba28\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"
Sep 11 21:31:37.098734 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.098615    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4pmkk\" (UID: \"d08eb046-1dfa-43cf-a851-0b722e18ba28\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"
Sep 11 21:31:37.098734 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.098651    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrkrz\" (UniqueName: \"kubernetes.io/projected/d08eb046-1dfa-43cf-a851-0b722e18ba28-kube-api-access-xrkrz\") pod \"cluster-monitoring-operator-6cd954b897-4pmkk\" (UID: \"d08eb046-1dfa-43cf-a851-0b722e18ba28\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"
Sep 11 21:31:37.099840 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.099798    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7448bd7fc8-p7xdg"]
Sep 11 21:31:37.101781 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.101766    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:37.106627 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.106604    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 11 21:31:37.107017 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.107001    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 11 21:31:37.108360 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.108341    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 11 21:31:37.110558 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.110542    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dqdwr\""
Sep 11 21:31:37.110716 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.110704    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 11 21:31:37.111100 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.111084    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 21:31:37.113903 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.113888    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 21:31:37.131598 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.131569    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7448bd7fc8-p7xdg"]
Sep 11 21:31:37.160192 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.160158    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx"]
Sep 11 21:31:37.193457 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.193424    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b"]
Sep 11 21:31:37.195357 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.195341    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b"
Sep 11 21:31:37.197893 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.197872    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 11 21:31:37.198161 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.198132    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5tqnc\""
Sep 11 21:31:37.198326 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.198307    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 11 21:31:37.198449 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.198433    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:31:37.198964 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.198946    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrkrz\" (UniqueName: \"kubernetes.io/projected/d08eb046-1dfa-43cf-a851-0b722e18ba28-kube-api-access-xrkrz\") pod \"cluster-monitoring-operator-6cd954b897-4pmkk\" (UID: \"d08eb046-1dfa-43cf-a851-0b722e18ba28\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"
Sep 11 21:31:37.199047 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.198982    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-stats-auth\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:37.199047 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.199031    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-4sjgx\" (UID: \"545c8784-8a4a-433e-a576-4df8c2ec80e7\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx"
Sep 11 21:31:37.199146 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.199130    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:37.199193 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.199175    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqcz2\" (UniqueName: \"kubernetes.io/projected/162606cf-63cc-456f-89f4-5d222f5bb4f8-kube-api-access-qqcz2\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:37.199292 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.199277    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d08eb046-1dfa-43cf-a851-0b722e18ba28-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-4pmkk\" (UID: \"d08eb046-1dfa-43cf-a851-0b722e18ba28\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"
Sep 11 21:31:37.199331 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.199301    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4pmkk\" (UID: \"d08eb046-1dfa-43cf-a851-0b722e18ba28\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"
Sep 11 21:31:37.199331 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.199321    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-default-certificate\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:37.199414 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.199345    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:37.199414 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.199372    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/545c8784-8a4a-433e-a576-4df8c2ec80e7-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-4sjgx\" (UID: \"545c8784-8a4a-433e-a576-4df8c2ec80e7\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx"
Sep 11 21:31:37.199486 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:37.199426    2544 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 21:31:37.199526 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:37.199501    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls podName:d08eb046-1dfa-43cf-a851-0b722e18ba28 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:37.699480644 +0000 UTC m=+60.668897446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-4pmkk" (UID: "d08eb046-1dfa-43cf-a851-0b722e18ba28") : secret "cluster-monitoring-operator-tls" not found
Sep 11 21:31:37.199944 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.199927    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d08eb046-1dfa-43cf-a851-0b722e18ba28-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-4pmkk\" (UID: \"d08eb046-1dfa-43cf-a851-0b722e18ba28\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"
Sep 11 21:31:37.211026 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.211000    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b"]
Sep 11 21:31:37.221278 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.221247    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrkrz\" (UniqueName: \"kubernetes.io/projected/d08eb046-1dfa-43cf-a851-0b722e18ba28-kube-api-access-xrkrz\") pod \"cluster-monitoring-operator-6cd954b897-4pmkk\" (UID: \"d08eb046-1dfa-43cf-a851-0b722e18ba28\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"
Sep 11 21:31:37.300142 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.300101    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-stats-auth\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:37.300142 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.300145    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdhm7\" (UniqueName: \"kubernetes.io/projected/9a99a7a9-f2be-43bd-943d-e634ae8ad980-kube-api-access-fdhm7\") pod \"cluster-samples-operator-86499c89c4-lcj2b\" (UID: \"9a99a7a9-f2be-43bd-943d-e634ae8ad980\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b"
Sep 11 21:31:37.300482 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.300188    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-4sjgx\" (UID: \"545c8784-8a4a-433e-a576-4df8c2ec80e7\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx"
Sep 11 21:31:37.300482 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.300273    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:37.300482 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.300300    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqcz2\" (UniqueName: \"kubernetes.io/projected/162606cf-63cc-456f-89f4-5d222f5bb4f8-kube-api-access-qqcz2\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:37.300482 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.300319    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a99a7a9-f2be-43bd-943d-e634ae8ad980-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lcj2b\" (UID: \"9a99a7a9-f2be-43bd-943d-e634ae8ad980\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b"
Sep 11 21:31:37.300482 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:37.300332    2544 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 21:31:37.300482 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.300378    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-default-certificate\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:37.300482 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:37.300403    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert podName:545c8784-8a4a-433e-a576-4df8c2ec80e7 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:37.800383055 +0000 UTC m=+60.769799853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-4sjgx" (UID: "545c8784-8a4a-433e-a576-4df8c2ec80e7") : secret "networking-console-plugin-cert" not found
Sep 11 21:31:37.300482 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:37.300403    2544 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 21:31:37.300860 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.300493    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:37.300860 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:37.300536    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs podName:162606cf-63cc-456f-89f4-5d222f5bb4f8 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:37.800518305 +0000 UTC m=+60.769935114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs") pod "router-default-7448bd7fc8-p7xdg" (UID: "162606cf-63cc-456f-89f4-5d222f5bb4f8") : secret "router-metrics-certs-default" not found
Sep 11 21:31:37.300860 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.300556    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/545c8784-8a4a-433e-a576-4df8c2ec80e7-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-4sjgx\" (UID: \"545c8784-8a4a-433e-a576-4df8c2ec80e7\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx"
Sep 11 21:31:37.300860 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:37.300566    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle podName:162606cf-63cc-456f-89f4-5d222f5bb4f8 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:37.80055754 +0000 UTC m=+60.769974338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle") pod "router-default-7448bd7fc8-p7xdg" (UID: "162606cf-63cc-456f-89f4-5d222f5bb4f8") : configmap references non-existent config key: service-ca.crt
Sep 11 21:31:37.301095 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.301077    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/545c8784-8a4a-433e-a576-4df8c2ec80e7-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-4sjgx\" (UID: \"545c8784-8a4a-433e-a576-4df8c2ec80e7\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx"
Sep 11 21:31:37.302708 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.302685    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-stats-auth\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:37.302799 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.302788    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-default-certificate\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:37.313177 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.313146    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqcz2\" (UniqueName: \"kubernetes.io/projected/162606cf-63cc-456f-89f4-5d222f5bb4f8-kube-api-access-qqcz2\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:37.401750 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.401659    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdhm7\" (UniqueName: \"kubernetes.io/projected/9a99a7a9-f2be-43bd-943d-e634ae8ad980-kube-api-access-fdhm7\") pod \"cluster-samples-operator-86499c89c4-lcj2b\" (UID: \"9a99a7a9-f2be-43bd-943d-e634ae8ad980\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b"
Sep 11 21:31:37.401920 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.401828    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a99a7a9-f2be-43bd-943d-e634ae8ad980-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lcj2b\" (UID: \"9a99a7a9-f2be-43bd-943d-e634ae8ad980\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b"
Sep 11 21:31:37.401983 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:37.401934    2544 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 21:31:37.402032 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:37.401991    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a99a7a9-f2be-43bd-943d-e634ae8ad980-samples-operator-tls podName:9a99a7a9-f2be-43bd-943d-e634ae8ad980 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:37.901975621 +0000 UTC m=+60.871392421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9a99a7a9-f2be-43bd-943d-e634ae8ad980-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-lcj2b" (UID: "9a99a7a9-f2be-43bd-943d-e634ae8ad980") : secret "samples-operator-tls" not found
Sep 11 21:31:37.411744 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.411711    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdhm7\" (UniqueName: \"kubernetes.io/projected/9a99a7a9-f2be-43bd-943d-e634ae8ad980-kube-api-access-fdhm7\") pod \"cluster-samples-operator-86499c89c4-lcj2b\" (UID: \"9a99a7a9-f2be-43bd-943d-e634ae8ad980\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b"
Sep 11 21:31:37.703958 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.703850    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4pmkk\" (UID: \"d08eb046-1dfa-43cf-a851-0b722e18ba28\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"
Sep 11 21:31:37.704153 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:37.704025    2544 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 21:31:37.704153 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:37.704117    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls podName:d08eb046-1dfa-43cf-a851-0b722e18ba28 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:38.704094502 +0000 UTC m=+61.673511300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-4pmkk" (UID: "d08eb046-1dfa-43cf-a851-0b722e18ba28") : secret "cluster-monitoring-operator-tls" not found
Sep 11 21:31:37.804705 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.804663    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:37.805137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.804738    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-4sjgx\" (UID: \"545c8784-8a4a-433e-a576-4df8c2ec80e7\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx"
Sep 11 21:31:37.805137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.804794    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:37.805137 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:37.804834    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle podName:162606cf-63cc-456f-89f4-5d222f5bb4f8 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:38.804813007 +0000 UTC m=+61.774229813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle") pod "router-default-7448bd7fc8-p7xdg" (UID: "162606cf-63cc-456f-89f4-5d222f5bb4f8") : configmap references non-existent config key: service-ca.crt
Sep 11 21:31:37.805137 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:37.804877    2544 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 21:31:37.805137 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:37.804891    2544 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 21:31:37.805137 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:37.804939    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert podName:545c8784-8a4a-433e-a576-4df8c2ec80e7 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:38.804926808 +0000 UTC m=+61.774343610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-4sjgx" (UID: "545c8784-8a4a-433e-a576-4df8c2ec80e7") : secret "networking-console-plugin-cert" not found
Sep 11 21:31:37.805137 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:37.804961    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs podName:162606cf-63cc-456f-89f4-5d222f5bb4f8 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:38.804954379 +0000 UTC m=+61.774371176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs") pod "router-default-7448bd7fc8-p7xdg" (UID: "162606cf-63cc-456f-89f4-5d222f5bb4f8") : secret "router-metrics-certs-default" not found
Sep 11 21:31:37.905741 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:37.905698    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a99a7a9-f2be-43bd-943d-e634ae8ad980-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lcj2b\" (UID: \"9a99a7a9-f2be-43bd-943d-e634ae8ad980\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b"
Sep 11 21:31:37.905905 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:37.905813    2544 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 21:31:37.905905 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:37.905875    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a99a7a9-f2be-43bd-943d-e634ae8ad980-samples-operator-tls podName:9a99a7a9-f2be-43bd-943d-e634ae8ad980 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:38.905861628 +0000 UTC m=+61.875278439 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9a99a7a9-f2be-43bd-943d-e634ae8ad980-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-lcj2b" (UID: "9a99a7a9-f2be-43bd-943d-e634ae8ad980") : secret "samples-operator-tls" not found
Sep 11 21:31:38.710700 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:38.710661    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4pmkk\" (UID: \"d08eb046-1dfa-43cf-a851-0b722e18ba28\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"
Sep 11 21:31:38.710908 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:38.710827    2544 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 21:31:38.710908 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:38.710903    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls podName:d08eb046-1dfa-43cf-a851-0b722e18ba28 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:40.710884564 +0000 UTC m=+63.680301364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-4pmkk" (UID: "d08eb046-1dfa-43cf-a851-0b722e18ba28") : secret "cluster-monitoring-operator-tls" not found
Sep 11 21:31:38.811027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:38.810994    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:38.811391 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:38.811072    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:38.811391 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:38.811131    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-4sjgx\" (UID: \"545c8784-8a4a-433e-a576-4df8c2ec80e7\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx"
Sep 11 21:31:38.811391 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:38.811159    2544 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 21:31:38.811391 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:38.811239    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs podName:162606cf-63cc-456f-89f4-5d222f5bb4f8 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:40.811204919 +0000 UTC m=+63.780621720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs") pod "router-default-7448bd7fc8-p7xdg" (UID: "162606cf-63cc-456f-89f4-5d222f5bb4f8") : secret "router-metrics-certs-default" not found
Sep 11 21:31:38.811391 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:38.811258    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle podName:162606cf-63cc-456f-89f4-5d222f5bb4f8 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:40.81124857 +0000 UTC m=+63.780665374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle") pod "router-default-7448bd7fc8-p7xdg" (UID: "162606cf-63cc-456f-89f4-5d222f5bb4f8") : configmap references non-existent config key: service-ca.crt
Sep 11 21:31:38.811391 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:38.811271    2544 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 21:31:38.811391 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:38.811310    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert podName:545c8784-8a4a-433e-a576-4df8c2ec80e7 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:40.811300938 +0000 UTC m=+63.780717750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-4sjgx" (UID: "545c8784-8a4a-433e-a576-4df8c2ec80e7") : secret "networking-console-plugin-cert" not found
Sep 11 21:31:38.911770 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:38.911731    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a99a7a9-f2be-43bd-943d-e634ae8ad980-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lcj2b\" (UID: \"9a99a7a9-f2be-43bd-943d-e634ae8ad980\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b"
Sep 11 21:31:38.911941 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:38.911897    2544 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 21:31:38.911981 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:38.911966    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a99a7a9-f2be-43bd-943d-e634ae8ad980-samples-operator-tls podName:9a99a7a9-f2be-43bd-943d-e634ae8ad980 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:40.911946288 +0000 UTC m=+63.881363088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9a99a7a9-f2be-43bd-943d-e634ae8ad980-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-lcj2b" (UID: "9a99a7a9-f2be-43bd-943d-e634ae8ad980") : secret "samples-operator-tls" not found
Sep 11 21:31:40.726556 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:40.726513    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4pmkk\" (UID: \"d08eb046-1dfa-43cf-a851-0b722e18ba28\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"
Sep 11 21:31:40.726954 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:40.726667    2544 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 21:31:40.726954 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:40.726736    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls podName:d08eb046-1dfa-43cf-a851-0b722e18ba28 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:44.726721697 +0000 UTC m=+67.696138495 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-4pmkk" (UID: "d08eb046-1dfa-43cf-a851-0b722e18ba28") : secret "cluster-monitoring-operator-tls" not found
Sep 11 21:31:40.827724 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:40.827688    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:40.827879 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:40.827749    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-4sjgx\" (UID: \"545c8784-8a4a-433e-a576-4df8c2ec80e7\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx"
Sep 11 21:31:40.827879 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:40.827790    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:40.827879 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:40.827864    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle podName:162606cf-63cc-456f-89f4-5d222f5bb4f8 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:44.82784653 +0000 UTC m=+67.797263332 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle") pod "router-default-7448bd7fc8-p7xdg" (UID: "162606cf-63cc-456f-89f4-5d222f5bb4f8") : configmap references non-existent config key: service-ca.crt
Sep 11 21:31:40.828039 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:40.827885    2544 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 21:31:40.828039 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:40.827911    2544 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 21:31:40.828039 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:40.827921    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs podName:162606cf-63cc-456f-89f4-5d222f5bb4f8 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:44.827911559 +0000 UTC m=+67.797328357 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs") pod "router-default-7448bd7fc8-p7xdg" (UID: "162606cf-63cc-456f-89f4-5d222f5bb4f8") : secret "router-metrics-certs-default" not found
Sep 11 21:31:40.828039 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:40.827982    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert podName:545c8784-8a4a-433e-a576-4df8c2ec80e7 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:44.827963323 +0000 UTC m=+67.797380128 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-4sjgx" (UID: "545c8784-8a4a-433e-a576-4df8c2ec80e7") : secret "networking-console-plugin-cert" not found
Sep 11 21:31:40.928685 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:40.928650    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a99a7a9-f2be-43bd-943d-e634ae8ad980-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lcj2b\" (UID: \"9a99a7a9-f2be-43bd-943d-e634ae8ad980\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b"
Sep 11 21:31:40.928863 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:40.928807    2544 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 21:31:40.928904 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:40.928870    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a99a7a9-f2be-43bd-943d-e634ae8ad980-samples-operator-tls podName:9a99a7a9-f2be-43bd-943d-e634ae8ad980 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:44.928856181 +0000 UTC m=+67.898272979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9a99a7a9-f2be-43bd-943d-e634ae8ad980-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-lcj2b" (UID: "9a99a7a9-f2be-43bd-943d-e634ae8ad980") : secret "samples-operator-tls" not found
Sep 11 21:31:42.498260 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.498210    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2qg7x"]
Sep 11 21:31:42.526596 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.526562    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2qg7x"]
Sep 11 21:31:42.526751 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.526684    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:42.529207 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.529177    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 21:31:42.529332 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.529246    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 21:31:42.529446 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.529387    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 21:31:42.529591 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.529573    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 21:31:42.529635 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.529594    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hpqdp\""
Sep 11 21:31:42.541792 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.541765    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92e79e38-c302-421c-886d-a0e68872587d-crio-socket\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:42.541932 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.541813    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:42.541932 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.541883    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92e79e38-c302-421c-886d-a0e68872587d-data-volume\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:42.541932 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.541907    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsb5s\" (UniqueName: \"kubernetes.io/projected/92e79e38-c302-421c-886d-a0e68872587d-kube-api-access-tsb5s\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:42.542063 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.541934    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92e79e38-c302-421c-886d-a0e68872587d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:42.643024 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.642978    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92e79e38-c302-421c-886d-a0e68872587d-crio-socket\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:42.643024 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.643028    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:42.643275 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.643108    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92e79e38-c302-421c-886d-a0e68872587d-crio-socket\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:42.643275 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:42.643119    2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:42.643275 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.643156    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92e79e38-c302-421c-886d-a0e68872587d-data-volume\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:42.643275 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.643184    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsb5s\" (UniqueName: \"kubernetes.io/projected/92e79e38-c302-421c-886d-a0e68872587d-kube-api-access-tsb5s\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:42.643275 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:42.643199    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls podName:92e79e38-c302-421c-886d-a0e68872587d nodeName:}" failed. No retries permitted until 2026-09-11 21:31:43.143180227 +0000 UTC m=+66.112597040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qg7x" (UID: "92e79e38-c302-421c-886d-a0e68872587d") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:42.643275 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.643256    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92e79e38-c302-421c-886d-a0e68872587d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:42.643525 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:42.643334    2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:31:42.643525 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:42.643378    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92e79e38-c302-421c-886d-a0e68872587d-kube-rbac-proxy-cm podName:92e79e38-c302-421c-886d-a0e68872587d nodeName:}" failed. No retries permitted until 2026-09-11 21:31:43.143365478 +0000 UTC m=+66.112782276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92e79e38-c302-421c-886d-a0e68872587d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qg7x" (UID: "92e79e38-c302-421c-886d-a0e68872587d") : configmap "kube-rbac-proxy" not found
Sep 11 21:31:42.643596 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.643563    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92e79e38-c302-421c-886d-a0e68872587d-data-volume\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:42.653155 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:42.653137    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsb5s\" (UniqueName: \"kubernetes.io/projected/92e79e38-c302-421c-886d-a0e68872587d-kube-api-access-tsb5s\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:43.147619 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.147577    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92e79e38-c302-421c-886d-a0e68872587d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:43.147822 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:43.147697    2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:31:43.147822 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.147709    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:31:43.147822 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:43.147771    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92e79e38-c302-421c-886d-a0e68872587d-kube-rbac-proxy-cm podName:92e79e38-c302-421c-886d-a0e68872587d nodeName:}" failed. No retries permitted until 2026-09-11 21:31:44.1477547 +0000 UTC m=+67.117171520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92e79e38-c302-421c-886d-a0e68872587d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qg7x" (UID: "92e79e38-c302-421c-886d-a0e68872587d") : configmap "kube-rbac-proxy" not found
Sep 11 21:31:43.147822 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.147803    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:43.148026 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:43.147839    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:31:43.148026 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:43.147899    2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:43.148026 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:43.147904    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls podName:77e62236-28b9-4261-85cb-b2aba118874b nodeName:}" failed. No retries permitted until 2026-09-11 21:32:15.14788695 +0000 UTC m=+98.117303765 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls") pod "dns-default-pnzdj" (UID: "77e62236-28b9-4261-85cb-b2aba118874b") : secret "dns-default-metrics-tls" not found
Sep 11 21:31:43.148026 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:43.147926    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls podName:92e79e38-c302-421c-886d-a0e68872587d nodeName:}" failed. No retries permitted until 2026-09-11 21:31:44.147918218 +0000 UTC m=+67.117335016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qg7x" (UID: "92e79e38-c302-421c-886d-a0e68872587d") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:43.249148 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.249105    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs\") pod \"network-metrics-daemon-k94pz\" (UID: \"047e6301-db2b-4c57-86ad-20ca80ff019a\") " pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:31:43.249370 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.249264    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret\") pod \"global-pull-secret-syncer-fjvb6\" (UID: \"fef5eac9-cf6d-40b0-9bb2-374516793dc2\") " pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:31:43.251511 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.251490    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 21:31:43.251663 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.251646    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 21:31:43.259396 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:43.259372    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 21:31:43.259472 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:43.259452    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs podName:047e6301-db2b-4c57-86ad-20ca80ff019a nodeName:}" failed. No retries permitted until 2026-09-11 21:32:47.25942755 +0000 UTC m=+130.228844357 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs") pod "network-metrics-daemon-k94pz" (UID: "047e6301-db2b-4c57-86ad-20ca80ff019a") : secret "metrics-daemon-secret" not found
Sep 11 21:31:43.261612 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.261583    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fef5eac9-cf6d-40b0-9bb2-374516793dc2-original-pull-secret\") pod \"global-pull-secret-syncer-fjvb6\" (UID: \"fef5eac9-cf6d-40b0-9bb2-374516793dc2\") " pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:31:43.349905 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.349866    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl6th\" (UniqueName: \"kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th\") pod \"network-check-target-wmjnx\" (UID: \"6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b\") " pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:31:43.352331 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.352311    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 21:31:43.362343 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.362318    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 21:31:43.373663 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.373636    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl6th\" (UniqueName: \"kubernetes.io/projected/6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b-kube-api-access-zl6th\") pod \"network-check-target-wmjnx\" (UID: \"6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b\") " pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:31:43.397736 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.397657    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wxgvf\""
Sep 11 21:31:43.406103 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.406080    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:31:43.497524 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.497475    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjvb6"
Sep 11 21:31:43.674408 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.674330    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wmjnx"]
Sep 11 21:31:43.677606 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:31:43.677576    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f32fbc4_b1f0_46de_94d4_9a5eaef08d3b.slice/crio-8b5cc4b197cb32b0cc140f02cd936b5f3fefe012a370912a78b8db1e6cd4a014 WatchSource:0}: Error finding container 8b5cc4b197cb32b0cc140f02cd936b5f3fefe012a370912a78b8db1e6cd4a014: Status 404 returned error can't find the container with id 8b5cc4b197cb32b0cc140f02cd936b5f3fefe012a370912a78b8db1e6cd4a014
Sep 11 21:31:43.763171 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.763142    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fjvb6"]
Sep 11 21:31:43.766702 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:31:43.766674    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef5eac9_cf6d_40b0_9bb2_374516793dc2.slice/crio-2a07c390716a050b364cb0f575185410a8b7ca887c971d3391f3e5f900437919 WatchSource:0}: Error finding container 2a07c390716a050b364cb0f575185410a8b7ca887c971d3391f3e5f900437919: Status 404 returned error can't find the container with id 2a07c390716a050b364cb0f575185410a8b7ca887c971d3391f3e5f900437919
Sep 11 21:31:43.894064 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.894019    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fjvb6" event={"ID":"fef5eac9-cf6d-40b0-9bb2-374516793dc2","Type":"ContainerStarted","Data":"2a07c390716a050b364cb0f575185410a8b7ca887c971d3391f3e5f900437919"}
Sep 11 21:31:43.895162 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:43.895141    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wmjnx" event={"ID":"6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b","Type":"ContainerStarted","Data":"8b5cc4b197cb32b0cc140f02cd936b5f3fefe012a370912a78b8db1e6cd4a014"}
Sep 11 21:31:44.156574 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:44.156522    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:44.156765 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:44.156588    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92e79e38-c302-421c-886d-a0e68872587d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:44.156765 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:44.156677    2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:44.156765 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:44.156697    2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:31:44.156765 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:44.156761    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls podName:92e79e38-c302-421c-886d-a0e68872587d nodeName:}" failed. No retries permitted until 2026-09-11 21:31:46.156739666 +0000 UTC m=+69.126156465 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qg7x" (UID: "92e79e38-c302-421c-886d-a0e68872587d") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:44.156939 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:44.156784    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92e79e38-c302-421c-886d-a0e68872587d-kube-rbac-proxy-cm podName:92e79e38-c302-421c-886d-a0e68872587d nodeName:}" failed. No retries permitted until 2026-09-11 21:31:46.156771897 +0000 UTC m=+69.126188701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92e79e38-c302-421c-886d-a0e68872587d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qg7x" (UID: "92e79e38-c302-421c-886d-a0e68872587d") : configmap "kube-rbac-proxy" not found
Sep 11 21:31:44.762208 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:44.762173    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4pmkk\" (UID: \"d08eb046-1dfa-43cf-a851-0b722e18ba28\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"
Sep 11 21:31:44.762634 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:44.762332    2544 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 21:31:44.762634 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:44.762404    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls podName:d08eb046-1dfa-43cf-a851-0b722e18ba28 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:52.762387557 +0000 UTC m=+75.731804364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-4pmkk" (UID: "d08eb046-1dfa-43cf-a851-0b722e18ba28") : secret "cluster-monitoring-operator-tls" not found
Sep 11 21:31:44.862887 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:44.862844    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-4sjgx\" (UID: \"545c8784-8a4a-433e-a576-4df8c2ec80e7\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx"
Sep 11 21:31:44.863087 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:44.862934    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:44.863087 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:44.863001    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:44.863087 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:44.863024    2544 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 21:31:44.863280 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:44.863100    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert podName:545c8784-8a4a-433e-a576-4df8c2ec80e7 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:52.863079722 +0000 UTC m=+75.832496525 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-4sjgx" (UID: "545c8784-8a4a-433e-a576-4df8c2ec80e7") : secret "networking-console-plugin-cert" not found
Sep 11 21:31:44.863280 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:44.863140    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle podName:162606cf-63cc-456f-89f4-5d222f5bb4f8 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:52.863121917 +0000 UTC m=+75.832538732 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle") pod "router-default-7448bd7fc8-p7xdg" (UID: "162606cf-63cc-456f-89f4-5d222f5bb4f8") : configmap references non-existent config key: service-ca.crt
Sep 11 21:31:44.863280 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:44.863167    2544 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 21:31:44.863280 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:44.863214    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs podName:162606cf-63cc-456f-89f4-5d222f5bb4f8 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:52.8632007 +0000 UTC m=+75.832617515 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs") pod "router-default-7448bd7fc8-p7xdg" (UID: "162606cf-63cc-456f-89f4-5d222f5bb4f8") : secret "router-metrics-certs-default" not found
Sep 11 21:31:44.963774 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:44.963730    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a99a7a9-f2be-43bd-943d-e634ae8ad980-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lcj2b\" (UID: \"9a99a7a9-f2be-43bd-943d-e634ae8ad980\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b"
Sep 11 21:31:44.963955 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:44.963909    2544 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 21:31:44.964020 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:44.963994    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a99a7a9-f2be-43bd-943d-e634ae8ad980-samples-operator-tls podName:9a99a7a9-f2be-43bd-943d-e634ae8ad980 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:52.963973561 +0000 UTC m=+75.933390381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9a99a7a9-f2be-43bd-943d-e634ae8ad980-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-lcj2b" (UID: "9a99a7a9-f2be-43bd-943d-e634ae8ad980") : secret "samples-operator-tls" not found
Sep 11 21:31:45.134835 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.134757    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6gf57_07c49e47-3378-4c6c-984a-802481fbf22c/dns-node-resolver/0.log"
Sep 11 21:31:45.218692 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.218657    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-6mhxs"]
Sep 11 21:31:45.221058 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.221031    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-6mhxs"
Sep 11 21:31:45.223452 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.223426    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 21:31:45.223573 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.223502    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ll9c8\""
Sep 11 21:31:45.224494 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.224472    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 21:31:45.224660 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.224478    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 21:31:45.225156 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.225007    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 21:31:45.229464 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.229444    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-6mhxs"]
Sep 11 21:31:45.266657 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.266621    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qgvf\" (UniqueName: \"kubernetes.io/projected/4be2de8e-5199-4de5-9662-9758d908aaac-kube-api-access-5qgvf\") pod \"service-ca-568869fdc5-6mhxs\" (UID: \"4be2de8e-5199-4de5-9662-9758d908aaac\") " pod="openshift-service-ca/service-ca-568869fdc5-6mhxs"
Sep 11 21:31:45.266844 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.266749    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4be2de8e-5199-4de5-9662-9758d908aaac-signing-key\") pod \"service-ca-568869fdc5-6mhxs\" (UID: \"4be2de8e-5199-4de5-9662-9758d908aaac\") " pod="openshift-service-ca/service-ca-568869fdc5-6mhxs"
Sep 11 21:31:45.266844 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.266777    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4be2de8e-5199-4de5-9662-9758d908aaac-signing-cabundle\") pod \"service-ca-568869fdc5-6mhxs\" (UID: \"4be2de8e-5199-4de5-9662-9758d908aaac\") " pod="openshift-service-ca/service-ca-568869fdc5-6mhxs"
Sep 11 21:31:45.368324 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.368284    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4be2de8e-5199-4de5-9662-9758d908aaac-signing-key\") pod \"service-ca-568869fdc5-6mhxs\" (UID: \"4be2de8e-5199-4de5-9662-9758d908aaac\") " pod="openshift-service-ca/service-ca-568869fdc5-6mhxs"
Sep 11 21:31:45.368324 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.368329    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4be2de8e-5199-4de5-9662-9758d908aaac-signing-cabundle\") pod \"service-ca-568869fdc5-6mhxs\" (UID: \"4be2de8e-5199-4de5-9662-9758d908aaac\") " pod="openshift-service-ca/service-ca-568869fdc5-6mhxs"
Sep 11 21:31:45.368556 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.368442    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qgvf\" (UniqueName: \"kubernetes.io/projected/4be2de8e-5199-4de5-9662-9758d908aaac-kube-api-access-5qgvf\") pod \"service-ca-568869fdc5-6mhxs\" (UID: \"4be2de8e-5199-4de5-9662-9758d908aaac\") " pod="openshift-service-ca/service-ca-568869fdc5-6mhxs"
Sep 11 21:31:45.369401 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.369377    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4be2de8e-5199-4de5-9662-9758d908aaac-signing-cabundle\") pod \"service-ca-568869fdc5-6mhxs\" (UID: \"4be2de8e-5199-4de5-9662-9758d908aaac\") " pod="openshift-service-ca/service-ca-568869fdc5-6mhxs"
Sep 11 21:31:45.371373 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.371345    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4be2de8e-5199-4de5-9662-9758d908aaac-signing-key\") pod \"service-ca-568869fdc5-6mhxs\" (UID: \"4be2de8e-5199-4de5-9662-9758d908aaac\") " pod="openshift-service-ca/service-ca-568869fdc5-6mhxs"
Sep 11 21:31:45.380107 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.380076    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qgvf\" (UniqueName: \"kubernetes.io/projected/4be2de8e-5199-4de5-9662-9758d908aaac-kube-api-access-5qgvf\") pod \"service-ca-568869fdc5-6mhxs\" (UID: \"4be2de8e-5199-4de5-9662-9758d908aaac\") " pod="openshift-service-ca/service-ca-568869fdc5-6mhxs"
Sep 11 21:31:45.534599 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.534555    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-6mhxs"
Sep 11 21:31:45.822996 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.822965    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-6mhxs"]
Sep 11 21:31:45.825890 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:31:45.825839    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be2de8e_5199_4de5_9662_9758d908aaac.slice/crio-8b2530b5b958f9e07225029c35d68dfa9f616b6cb39003aa5801e62afb019671 WatchSource:0}: Error finding container 8b2530b5b958f9e07225029c35d68dfa9f616b6cb39003aa5801e62afb019671: Status 404 returned error can't find the container with id 8b2530b5b958f9e07225029c35d68dfa9f616b6cb39003aa5801e62afb019671
Sep 11 21:31:45.901735 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:45.901702    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-6mhxs" event={"ID":"4be2de8e-5199-4de5-9662-9758d908aaac","Type":"ContainerStarted","Data":"8b2530b5b958f9e07225029c35d68dfa9f616b6cb39003aa5801e62afb019671"}
Sep 11 21:31:46.176326 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:46.176239    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:46.176326 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:46.176289    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92e79e38-c302-421c-886d-a0e68872587d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:46.176506 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:46.176369    2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:31:46.176506 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:46.176418    2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:46.176506 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:46.176442    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92e79e38-c302-421c-886d-a0e68872587d-kube-rbac-proxy-cm podName:92e79e38-c302-421c-886d-a0e68872587d nodeName:}" failed. No retries permitted until 2026-09-11 21:31:50.176427187 +0000 UTC m=+73.145844006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92e79e38-c302-421c-886d-a0e68872587d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qg7x" (UID: "92e79e38-c302-421c-886d-a0e68872587d") : configmap "kube-rbac-proxy" not found
Sep 11 21:31:46.176626 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:46.176524    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls podName:92e79e38-c302-421c-886d-a0e68872587d nodeName:}" failed. No retries permitted until 2026-09-11 21:31:50.176505331 +0000 UTC m=+73.145922146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qg7x" (UID: "92e79e38-c302-421c-886d-a0e68872587d") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:46.535165 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:46.535137    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sh5wj_728ca319-dbd4-4ade-8a93-10191019f4df/node-ca/0.log"
Sep 11 21:31:47.136877 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:47.136846    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dprjf_ae83570c-f2ee-4671-818e-f68d8c4cd4d0/serve-healthcheck-canary/0.log"
Sep 11 21:31:48.912076 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:48.912029    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fjvb6" event={"ID":"fef5eac9-cf6d-40b0-9bb2-374516793dc2","Type":"ContainerStarted","Data":"0ccbeadd29d5f0d12a306da06201d851eacbd34224589df5993e413b8b173d0e"}
Sep 11 21:31:48.913867 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:48.913837    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wmjnx" event={"ID":"6f32fbc4-b1f0-46de-94d4-9a5eaef08d3b","Type":"ContainerStarted","Data":"4c363005741b54539a2fbb01480247e4cfc5510facfbd6d34f3f6bf29cb3c1c7"}
Sep 11 21:31:48.914000 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:48.913986    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:31:48.940354 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:48.940291    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fjvb6" podStartSLOduration=67.351144534 podStartE2EDuration="1m11.940270588s" totalImagesPullingTime="4.589126054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:48.939914445 +0000 UTC m=+71.909331268" watchObservedRunningTime="2026-09-11 21:31:48.940270588 +0000 UTC m=+71.909687410"
Sep 11 21:31:48.960771 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:48.960715    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wmjnx" podStartSLOduration=67.286523789 podStartE2EDuration="1m11.960697871s" totalImagesPullingTime="4.674174082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:48.960497965 +0000 UTC m=+71.929914784" watchObservedRunningTime="2026-09-11 21:31:48.960697871 +0000 UTC m=+71.930114693"
Sep 11 21:31:49.917877 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:49.917829    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-6mhxs" event={"ID":"4be2de8e-5199-4de5-9662-9758d908aaac","Type":"ContainerStarted","Data":"19b84a86bdf5bf42ed1006d81a1246ba2f582e0b198660d1a60a274073818584"}
Sep 11 21:31:49.933629 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:49.933577    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-6mhxs" podStartSLOduration=1.01294561 podStartE2EDuration="4.933555998s" totalImagesPullingTime="3.920610388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:31:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:49.933191655 +0000 UTC m=+72.902608477" watchObservedRunningTime="2026-09-11 21:31:49.933555998 +0000 UTC m=+72.902972818"
Sep 11 21:31:50.214343 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:50.214240    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:50.214343 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:50.214292    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92e79e38-c302-421c-886d-a0e68872587d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:50.214544 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:50.214446    2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:50.214544 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:50.214457    2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:31:50.214544 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:50.214506    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls podName:92e79e38-c302-421c-886d-a0e68872587d nodeName:}" failed. No retries permitted until 2026-09-11 21:31:58.214489802 +0000 UTC m=+81.183906603 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qg7x" (UID: "92e79e38-c302-421c-886d-a0e68872587d") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:50.214544 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:50.214531    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92e79e38-c302-421c-886d-a0e68872587d-kube-rbac-proxy-cm podName:92e79e38-c302-421c-886d-a0e68872587d nodeName:}" failed. No retries permitted until 2026-09-11 21:31:58.214524709 +0000 UTC m=+81.183941507 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92e79e38-c302-421c-886d-a0e68872587d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qg7x" (UID: "92e79e38-c302-421c-886d-a0e68872587d") : configmap "kube-rbac-proxy" not found
Sep 11 21:31:52.835731 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:52.835688    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4pmkk\" (UID: \"d08eb046-1dfa-43cf-a851-0b722e18ba28\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"
Sep 11 21:31:52.836125 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:52.835847    2544 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 21:31:52.836125 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:52.835917    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls podName:d08eb046-1dfa-43cf-a851-0b722e18ba28 nodeName:}" failed. No retries permitted until 2026-09-11 21:32:08.835901637 +0000 UTC m=+91.805318438 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-4pmkk" (UID: "d08eb046-1dfa-43cf-a851-0b722e18ba28") : secret "cluster-monitoring-operator-tls" not found
Sep 11 21:31:52.936710 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:52.936680    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-4sjgx\" (UID: \"545c8784-8a4a-433e-a576-4df8c2ec80e7\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx"
Sep 11 21:31:52.936864 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:52.936734    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:52.936864 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:52.936814    2544 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 21:31:52.936864 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:52.936819    2544 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 21:31:52.936864 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:52.936860    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs podName:162606cf-63cc-456f-89f4-5d222f5bb4f8 nodeName:}" failed. No retries permitted until 2026-09-11 21:32:08.936847732 +0000 UTC m=+91.906264530 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs") pod "router-default-7448bd7fc8-p7xdg" (UID: "162606cf-63cc-456f-89f4-5d222f5bb4f8") : secret "router-metrics-certs-default" not found
Sep 11 21:31:52.936998 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:52.936938    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert podName:545c8784-8a4a-433e-a576-4df8c2ec80e7 nodeName:}" failed. No retries permitted until 2026-09-11 21:32:08.936919027 +0000 UTC m=+91.906335835 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-4sjgx" (UID: "545c8784-8a4a-433e-a576-4df8c2ec80e7") : secret "networking-console-plugin-cert" not found
Sep 11 21:31:52.937035 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:52.937007    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:31:52.937132 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:52.937122    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle podName:162606cf-63cc-456f-89f4-5d222f5bb4f8 nodeName:}" failed. No retries permitted until 2026-09-11 21:32:08.937113915 +0000 UTC m=+91.906530712 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle") pod "router-default-7448bd7fc8-p7xdg" (UID: "162606cf-63cc-456f-89f4-5d222f5bb4f8") : configmap references non-existent config key: service-ca.crt
Sep 11 21:31:53.037938 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:53.037898    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a99a7a9-f2be-43bd-943d-e634ae8ad980-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lcj2b\" (UID: \"9a99a7a9-f2be-43bd-943d-e634ae8ad980\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b"
Sep 11 21:31:53.040262 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:53.040230    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a99a7a9-f2be-43bd-943d-e634ae8ad980-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lcj2b\" (UID: \"9a99a7a9-f2be-43bd-943d-e634ae8ad980\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b"
Sep 11 21:31:53.107143 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:53.107071    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5tqnc\""
Sep 11 21:31:53.115350 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:53.115320    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b"
Sep 11 21:31:53.383143 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:53.383021    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b"]
Sep 11 21:31:53.931280 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:53.931246    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b" event={"ID":"9a99a7a9-f2be-43bd-943d-e634ae8ad980","Type":"ContainerStarted","Data":"f6eea6a6b8d1b7a4232e7a275c2e7f4ddc10e5ffc7bf5379f11f0e96d739e9d7"}
Sep 11 21:31:55.940125 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:55.940087    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b" event={"ID":"9a99a7a9-f2be-43bd-943d-e634ae8ad980","Type":"ContainerStarted","Data":"8582a591a542e6484fadba126aec5490082877f17a5aea7e46311c275c5e7383"}
Sep 11 21:31:55.940601 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:55.940149    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b" event={"ID":"9a99a7a9-f2be-43bd-943d-e634ae8ad980","Type":"ContainerStarted","Data":"e5f5bbe9cf9b90cacdd5f1464b4d39876b3299e04ffd8ce54212aec89800609d"}
Sep 11 21:31:55.957385 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:55.957298    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lcj2b" podStartSLOduration=17.240827674 podStartE2EDuration="18.957281437s" totalImagesPullingTime="1.716453763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:55.956507185 +0000 UTC m=+78.925924005" watchObservedRunningTime="2026-09-11 21:31:55.957281437 +0000 UTC m=+78.926698251"
Sep 11 21:31:58.280815 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:58.280757    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92e79e38-c302-421c-886d-a0e68872587d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:58.281342 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:58.281022    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:31:58.281342 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:58.281132    2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:58.281342 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:31:58.281186    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls podName:92e79e38-c302-421c-886d-a0e68872587d nodeName:}" failed. No retries permitted until 2026-09-11 21:32:14.28117234 +0000 UTC m=+97.250589151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qg7x" (UID: "92e79e38-c302-421c-886d-a0e68872587d") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:58.281342 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:31:58.281296    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92e79e38-c302-421c-886d-a0e68872587d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:32:08.864483 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.864351    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4pmkk\" (UID: \"d08eb046-1dfa-43cf-a851-0b722e18ba28\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"
Sep 11 21:32:08.866717 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.866694    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d08eb046-1dfa-43cf-a851-0b722e18ba28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4pmkk\" (UID: \"d08eb046-1dfa-43cf-a851-0b722e18ba28\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"
Sep 11 21:32:08.965301 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.965262    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:32:08.965468 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.965323    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:32:08.965509 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.965480    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-4sjgx\" (UID: \"545c8784-8a4a-433e-a576-4df8c2ec80e7\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx"
Sep 11 21:32:08.965932 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.965909    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/162606cf-63cc-456f-89f4-5d222f5bb4f8-service-ca-bundle\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:32:08.967592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.967562    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162606cf-63cc-456f-89f4-5d222f5bb4f8-metrics-certs\") pod \"router-default-7448bd7fc8-p7xdg\" (UID: \"162606cf-63cc-456f-89f4-5d222f5bb4f8\") " pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:32:08.967845 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.967822    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/545c8784-8a4a-433e-a576-4df8c2ec80e7-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-4sjgx\" (UID: \"545c8784-8a4a-433e-a576-4df8c2ec80e7\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx"
Sep 11 21:32:08.985133 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.985098    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk"]
Sep 11 21:32:08.989505 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.989484    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"]
Sep 11 21:32:08.989651 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.989619    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk"
Sep 11 21:32:08.992401 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.992381    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 21:32:08.992520 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.992424    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 21:32:08.992577 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.992530    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 21:32:08.992686 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.992667    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:08.993207 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.993191    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 21:32:08.995231 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.995196    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 21:32:08.995337 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.995198    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 21:32:08.995607 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.995582    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 21:32:08.995828 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:08.995813    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 21:32:09.002684 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.002662    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk"]
Sep 11 21:32:09.003628 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.003609    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"]
Sep 11 21:32:09.066366 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.066329    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d1836612-bec8-49fc-9795-5b456135492b-ca\") pod \"cluster-proxy-proxy-agent-cc8cd66d-jpx6s\" (UID: \"d1836612-bec8-49fc-9795-5b456135492b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.066559 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.066398    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7389ecba-055b-4054-8198-a927e4aca379-klusterlet-config\") pod \"klusterlet-addon-workmgr-bf8db-4jgnk\" (UID: \"7389ecba-055b-4054-8198-a927e4aca379\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk"
Sep 11 21:32:09.066559 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.066473    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1836612-bec8-49fc-9795-5b456135492b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cc8cd66d-jpx6s\" (UID: \"d1836612-bec8-49fc-9795-5b456135492b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.066559 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.066535    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d1836612-bec8-49fc-9795-5b456135492b-hub\") pod \"cluster-proxy-proxy-agent-cc8cd66d-jpx6s\" (UID: \"d1836612-bec8-49fc-9795-5b456135492b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.066709 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.066583    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1836612-bec8-49fc-9795-5b456135492b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cc8cd66d-jpx6s\" (UID: \"d1836612-bec8-49fc-9795-5b456135492b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.066709 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.066627    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7389ecba-055b-4054-8198-a927e4aca379-tmp\") pod \"klusterlet-addon-workmgr-bf8db-4jgnk\" (UID: \"7389ecba-055b-4054-8198-a927e4aca379\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk"
Sep 11 21:32:09.066709 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.066689    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kdk2\" (UniqueName: \"kubernetes.io/projected/7389ecba-055b-4054-8198-a927e4aca379-kube-api-access-6kdk2\") pod \"klusterlet-addon-workmgr-bf8db-4jgnk\" (UID: \"7389ecba-055b-4054-8198-a927e4aca379\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk"
Sep 11 21:32:09.066810 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.066717    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d1836612-bec8-49fc-9795-5b456135492b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cc8cd66d-jpx6s\" (UID: \"d1836612-bec8-49fc-9795-5b456135492b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.066810 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.066735    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn62b\" (UniqueName: \"kubernetes.io/projected/d1836612-bec8-49fc-9795-5b456135492b-kube-api-access-tn62b\") pod \"cluster-proxy-proxy-agent-cc8cd66d-jpx6s\" (UID: \"d1836612-bec8-49fc-9795-5b456135492b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.098154 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.098120    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9242g\""
Sep 11 21:32:09.106229 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.106194    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"
Sep 11 21:32:09.167749 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.167702    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7389ecba-055b-4054-8198-a927e4aca379-klusterlet-config\") pod \"klusterlet-addon-workmgr-bf8db-4jgnk\" (UID: \"7389ecba-055b-4054-8198-a927e4aca379\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk"
Sep 11 21:32:09.167899 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.167780    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1836612-bec8-49fc-9795-5b456135492b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cc8cd66d-jpx6s\" (UID: \"d1836612-bec8-49fc-9795-5b456135492b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.167899 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.167819    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d1836612-bec8-49fc-9795-5b456135492b-hub\") pod \"cluster-proxy-proxy-agent-cc8cd66d-jpx6s\" (UID: \"d1836612-bec8-49fc-9795-5b456135492b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.167899 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.167875    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1836612-bec8-49fc-9795-5b456135492b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cc8cd66d-jpx6s\" (UID: \"d1836612-bec8-49fc-9795-5b456135492b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.168059 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.167911    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7389ecba-055b-4054-8198-a927e4aca379-tmp\") pod \"klusterlet-addon-workmgr-bf8db-4jgnk\" (UID: \"7389ecba-055b-4054-8198-a927e4aca379\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk"
Sep 11 21:32:09.168059 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.167954    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kdk2\" (UniqueName: \"kubernetes.io/projected/7389ecba-055b-4054-8198-a927e4aca379-kube-api-access-6kdk2\") pod \"klusterlet-addon-workmgr-bf8db-4jgnk\" (UID: \"7389ecba-055b-4054-8198-a927e4aca379\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk"
Sep 11 21:32:09.168059 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.167983    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d1836612-bec8-49fc-9795-5b456135492b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cc8cd66d-jpx6s\" (UID: \"d1836612-bec8-49fc-9795-5b456135492b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.168059 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.168007    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn62b\" (UniqueName: \"kubernetes.io/projected/d1836612-bec8-49fc-9795-5b456135492b-kube-api-access-tn62b\") pod \"cluster-proxy-proxy-agent-cc8cd66d-jpx6s\" (UID: \"d1836612-bec8-49fc-9795-5b456135492b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.168059 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.168043    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d1836612-bec8-49fc-9795-5b456135492b-ca\") pod \"cluster-proxy-proxy-agent-cc8cd66d-jpx6s\" (UID: \"d1836612-bec8-49fc-9795-5b456135492b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.168869 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.168532    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7389ecba-055b-4054-8198-a927e4aca379-tmp\") pod \"klusterlet-addon-workmgr-bf8db-4jgnk\" (UID: \"7389ecba-055b-4054-8198-a927e4aca379\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk"
Sep 11 21:32:09.169162 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.169137    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1836612-bec8-49fc-9795-5b456135492b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cc8cd66d-jpx6s\" (UID: \"d1836612-bec8-49fc-9795-5b456135492b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.171018 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.170987    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7389ecba-055b-4054-8198-a927e4aca379-klusterlet-config\") pod \"klusterlet-addon-workmgr-bf8db-4jgnk\" (UID: \"7389ecba-055b-4054-8198-a927e4aca379\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk"
Sep 11 21:32:09.171199 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.171174    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1836612-bec8-49fc-9795-5b456135492b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cc8cd66d-jpx6s\" (UID: \"d1836612-bec8-49fc-9795-5b456135492b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.171776 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.171730    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d1836612-bec8-49fc-9795-5b456135492b-hub\") pod \"cluster-proxy-proxy-agent-cc8cd66d-jpx6s\" (UID: \"d1836612-bec8-49fc-9795-5b456135492b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.173694 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.173640    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d1836612-bec8-49fc-9795-5b456135492b-ca\") pod \"cluster-proxy-proxy-agent-cc8cd66d-jpx6s\" (UID: \"d1836612-bec8-49fc-9795-5b456135492b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.175034 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.174730    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d1836612-bec8-49fc-9795-5b456135492b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cc8cd66d-jpx6s\" (UID: \"d1836612-bec8-49fc-9795-5b456135492b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.180498 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.180473    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kdk2\" (UniqueName: \"kubernetes.io/projected/7389ecba-055b-4054-8198-a927e4aca379-kube-api-access-6kdk2\") pod \"klusterlet-addon-workmgr-bf8db-4jgnk\" (UID: \"7389ecba-055b-4054-8198-a927e4aca379\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk"
Sep 11 21:32:09.180606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.180517    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn62b\" (UniqueName: \"kubernetes.io/projected/d1836612-bec8-49fc-9795-5b456135492b-kube-api-access-tn62b\") pod \"cluster-proxy-proxy-agent-cc8cd66d-jpx6s\" (UID: \"d1836612-bec8-49fc-9795-5b456135492b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.202972 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.202937    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8lxhs\""
Sep 11 21:32:09.211049 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.211022    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx"
Sep 11 21:32:09.212153 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.212135    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dqdwr\""
Sep 11 21:32:09.220064 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.220040    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:32:09.300647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.300549    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk"
Sep 11 21:32:09.319795 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.319764    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"
Sep 11 21:32:09.377758 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.377723    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk"]
Sep 11 21:32:09.382459 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:32:09.382428    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08eb046_1dfa_43cf_a851_0b722e18ba28.slice/crio-12ea4fe18dc28d37b9578a370afb528103969f65e600da3dddd22eb3a59c64ae WatchSource:0}: Error finding container 12ea4fe18dc28d37b9578a370afb528103969f65e600da3dddd22eb3a59c64ae: Status 404 returned error can't find the container with id 12ea4fe18dc28d37b9578a370afb528103969f65e600da3dddd22eb3a59c64ae
Sep 11 21:32:09.481965 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.481936    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx"]
Sep 11 21:32:09.485704 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:32:09.485674    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545c8784_8a4a_433e_a576_4df8c2ec80e7.slice/crio-5cf4bc1c32ab9a85e7359c6d1023e43bab365c037d5a57533daa6d07b5a62751 WatchSource:0}: Error finding container 5cf4bc1c32ab9a85e7359c6d1023e43bab365c037d5a57533daa6d07b5a62751: Status 404 returned error can't find the container with id 5cf4bc1c32ab9a85e7359c6d1023e43bab365c037d5a57533daa6d07b5a62751
Sep 11 21:32:09.504816 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.504787    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7448bd7fc8-p7xdg"]
Sep 11 21:32:09.506663 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:32:09.506636    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162606cf_63cc_456f_89f4_5d222f5bb4f8.slice/crio-272e64064eef858c8cd6ea4916aa8b8a11a5e5f86497662e188d523f5d626f77 WatchSource:0}: Error finding container 272e64064eef858c8cd6ea4916aa8b8a11a5e5f86497662e188d523f5d626f77: Status 404 returned error can't find the container with id 272e64064eef858c8cd6ea4916aa8b8a11a5e5f86497662e188d523f5d626f77
Sep 11 21:32:09.574369 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.574340    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk"]
Sep 11 21:32:09.577811 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:32:09.577784    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7389ecba_055b_4054_8198_a927e4aca379.slice/crio-5a6758a397f1b39aed483fbd4131c918c97ca097ccd514902ba06833d995a3da WatchSource:0}: Error finding container 5a6758a397f1b39aed483fbd4131c918c97ca097ccd514902ba06833d995a3da: Status 404 returned error can't find the container with id 5a6758a397f1b39aed483fbd4131c918c97ca097ccd514902ba06833d995a3da
Sep 11 21:32:09.600373 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.600344    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s"]
Sep 11 21:32:09.603410 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:32:09.603369    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1836612_bec8_49fc_9795_5b456135492b.slice/crio-859aa92dd07a7d2633e2dfc21cc99488fbca6304bf64a6ea28f7f4b38bbf445c WatchSource:0}: Error finding container 859aa92dd07a7d2633e2dfc21cc99488fbca6304bf64a6ea28f7f4b38bbf445c: Status 404 returned error can't find the container with id 859aa92dd07a7d2633e2dfc21cc99488fbca6304bf64a6ea28f7f4b38bbf445c
Sep 11 21:32:09.986868 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.986826    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk" event={"ID":"d08eb046-1dfa-43cf-a851-0b722e18ba28","Type":"ContainerStarted","Data":"12ea4fe18dc28d37b9578a370afb528103969f65e600da3dddd22eb3a59c64ae"}
Sep 11 21:32:09.987977 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.987955    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s" event={"ID":"d1836612-bec8-49fc-9795-5b456135492b","Type":"ContainerStarted","Data":"859aa92dd07a7d2633e2dfc21cc99488fbca6304bf64a6ea28f7f4b38bbf445c"}
Sep 11 21:32:09.989007 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.988987    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk" event={"ID":"7389ecba-055b-4054-8198-a927e4aca379","Type":"ContainerStarted","Data":"5a6758a397f1b39aed483fbd4131c918c97ca097ccd514902ba06833d995a3da"}
Sep 11 21:32:09.990370 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.990350    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7448bd7fc8-p7xdg" event={"ID":"162606cf-63cc-456f-89f4-5d222f5bb4f8","Type":"ContainerStarted","Data":"b0d0d901c86fd8a4d63828113afb07307d4c024d3fb6732a249e9df1debcec25"}
Sep 11 21:32:09.990479 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.990383    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7448bd7fc8-p7xdg" event={"ID":"162606cf-63cc-456f-89f4-5d222f5bb4f8","Type":"ContainerStarted","Data":"272e64064eef858c8cd6ea4916aa8b8a11a5e5f86497662e188d523f5d626f77"}
Sep 11 21:32:09.991578 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:09.991557    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx" event={"ID":"545c8784-8a4a-433e-a576-4df8c2ec80e7","Type":"ContainerStarted","Data":"5cf4bc1c32ab9a85e7359c6d1023e43bab365c037d5a57533daa6d07b5a62751"}
Sep 11 21:32:10.010847 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:10.010789    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7448bd7fc8-p7xdg" podStartSLOduration=33.010769036 podStartE2EDuration="33.010769036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:10.008637458 +0000 UTC m=+92.978054277" watchObservedRunningTime="2026-09-11 21:32:10.010769036 +0000 UTC m=+92.980185857"
Sep 11 21:32:10.221126 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:10.221077    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:32:10.224474 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:10.224445    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:32:10.997214 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:10.997134    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:32:10.999114 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:10.998909    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7448bd7fc8-p7xdg"
Sep 11 21:32:14.014618 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:14.014579    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s" event={"ID":"d1836612-bec8-49fc-9795-5b456135492b","Type":"ContainerStarted","Data":"226426614eb02d1db5bb809d6ee8362367909dda7054ae25343065d53fa06248"}
Sep 11 21:32:14.015061 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:14.014634    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s" event={"ID":"d1836612-bec8-49fc-9795-5b456135492b","Type":"ContainerStarted","Data":"61d508ac5d1c0d5da8e096ec21ddf3c35525a6712c97381d91452b6143d9093e"}
Sep 11 21:32:14.015061 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:14.014653    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s" event={"ID":"d1836612-bec8-49fc-9795-5b456135492b","Type":"ContainerStarted","Data":"27dfeff482a0c4d75efe2cef24745ecef5cfa3fca75b40ea255e04702e733fc2"}
Sep 11 21:32:14.016232 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:14.016189    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx" event={"ID":"545c8784-8a4a-433e-a576-4df8c2ec80e7","Type":"ContainerStarted","Data":"c907711a0537053cdbeefd8ca0cd00904b214093e7e1b5797d106915ebe5c28d"}
Sep 11 21:32:14.017746 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:14.017720    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk" event={"ID":"d08eb046-1dfa-43cf-a851-0b722e18ba28","Type":"ContainerStarted","Data":"d3c9599e28cb567c7f49404427df61a490d4b8cd0d06c5b1f4c59cdc9b3f35b6"}
Sep 11 21:32:14.034713 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:14.034666    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc8cd66d-jpx6s" podStartSLOduration=2.185812812 podStartE2EDuration="6.034650748s" totalImagesPullingTime="3.848837936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:14.03305533 +0000 UTC m=+97.002472154" watchObservedRunningTime="2026-09-11 21:32:14.034650748 +0000 UTC m=+97.004067567"
Sep 11 21:32:14.067915 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:14.067852    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sjgx" podStartSLOduration=33.170901375 podStartE2EDuration="37.067834947s" totalImagesPullingTime="3.896933572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:14.049384526 +0000 UTC m=+97.018801347" watchObservedRunningTime="2026-09-11 21:32:14.067834947 +0000 UTC m=+97.037251767"
Sep 11 21:32:14.068574 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:14.068546    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4pmkk" podStartSLOduration=34.069790362 podStartE2EDuration="38.068535609s" totalImagesPullingTime="3.998745247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:14.066988069 +0000 UTC m=+97.036404890" watchObservedRunningTime="2026-09-11 21:32:14.068535609 +0000 UTC m=+97.037952457"
Sep 11 21:32:14.319293 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:14.319168    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:32:14.321606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:14.321582    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92e79e38-c302-421c-886d-a0e68872587d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qg7x\" (UID: \"92e79e38-c302-421c-886d-a0e68872587d\") " pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:32:14.335513 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:14.335491    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2qg7x"
Sep 11 21:32:14.610998 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:14.610707    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2qg7x"]
Sep 11 21:32:14.613421 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:32:14.613398    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e79e38_c302_421c_886d_a0e68872587d.slice/crio-7348248d52ce5971c1ef726f348dae7c69cfff8e33cd02a4bbaf21f92b899133 WatchSource:0}: Error finding container 7348248d52ce5971c1ef726f348dae7c69cfff8e33cd02a4bbaf21f92b899133: Status 404 returned error can't find the container with id 7348248d52ce5971c1ef726f348dae7c69cfff8e33cd02a4bbaf21f92b899133
Sep 11 21:32:15.022568 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:15.022534    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2qg7x" event={"ID":"92e79e38-c302-421c-886d-a0e68872587d","Type":"ContainerStarted","Data":"e45ef6867923677ac07d4db9f2e80e2136ae2305e51fb9d9490f7a6d07a45cba"}
Sep 11 21:32:15.022930 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:15.022596    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2qg7x" event={"ID":"92e79e38-c302-421c-886d-a0e68872587d","Type":"ContainerStarted","Data":"7348248d52ce5971c1ef726f348dae7c69cfff8e33cd02a4bbaf21f92b899133"}
Sep 11 21:32:15.225716 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:15.225674    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:32:15.227942 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:15.227918    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77e62236-28b9-4261-85cb-b2aba118874b-metrics-tls\") pod \"dns-default-pnzdj\" (UID: \"77e62236-28b9-4261-85cb-b2aba118874b\") " pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:32:15.245883 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:15.245854    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-59g8b\""
Sep 11 21:32:15.253395 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:15.253373    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:32:15.524075 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:15.524049    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pnzdj"]
Sep 11 21:32:15.531330 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:32:15.531305    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e62236_28b9_4261_85cb_b2aba118874b.slice/crio-cb375137df0d641c8d6f9ecf07bbf165a02478fa20b14de2e7dde4dc3232fd84 WatchSource:0}: Error finding container cb375137df0d641c8d6f9ecf07bbf165a02478fa20b14de2e7dde4dc3232fd84: Status 404 returned error can't find the container with id cb375137df0d641c8d6f9ecf07bbf165a02478fa20b14de2e7dde4dc3232fd84
Sep 11 21:32:16.027535 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:16.027494    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pnzdj" event={"ID":"77e62236-28b9-4261-85cb-b2aba118874b","Type":"ContainerStarted","Data":"cb375137df0d641c8d6f9ecf07bbf165a02478fa20b14de2e7dde4dc3232fd84"}
Sep 11 21:32:16.029651 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:16.029616    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2qg7x" event={"ID":"92e79e38-c302-421c-886d-a0e68872587d","Type":"ContainerStarted","Data":"7eeb1a54a4930bad7179036bda024c3d3ab5604501093c516f0b51c6646882d5"}
Sep 11 21:32:18.038936 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:18.038895    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pnzdj" event={"ID":"77e62236-28b9-4261-85cb-b2aba118874b","Type":"ContainerStarted","Data":"18b6fc4ef2b39b352779b1d6a48db923036b5f2ba16944ba3d2d9f751c14ab76"}
Sep 11 21:32:18.039369 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:18.038956    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pnzdj" event={"ID":"77e62236-28b9-4261-85cb-b2aba118874b","Type":"ContainerStarted","Data":"c80acba3a0e23aeb3e21bfda724dc82caf16945086fad10740374443b3778791"}
Sep 11 21:32:18.039369 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:18.039023    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:32:18.040935 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:18.040906    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2qg7x" event={"ID":"92e79e38-c302-421c-886d-a0e68872587d","Type":"ContainerStarted","Data":"89f59456f137a61b7237fac8fd4085592e0c78772d20c8fbe00bcce287168570"}
Sep 11 21:32:18.070111 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:18.070051    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pnzdj" podStartSLOduration=65.129621008 podStartE2EDuration="1m7.070036627s" totalImagesPullingTime="1.940415619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:18.069639356 +0000 UTC m=+101.039056175" watchObservedRunningTime="2026-09-11 21:32:18.070036627 +0000 UTC m=+101.039453448"
Sep 11 21:32:19.920673 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:19.920630    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wmjnx"
Sep 11 21:32:19.939106 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:19.939054    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2qg7x" podStartSLOduration=35.217119393 podStartE2EDuration="37.939013828s" totalImagesPullingTime="2.721894435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:31:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:18.119070154 +0000 UTC m=+101.088487063" watchObservedRunningTime="2026-09-11 21:32:19.939013828 +0000 UTC m=+102.908430648"
Sep 11 21:32:24.352884 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.352846    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lttfq"]
Sep 11 21:32:24.355685 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.355659    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.359545 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.358510    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 21:32:24.359545 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.359178    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 21:32:24.359545 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.359410    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b8zbj\""
Sep 11 21:32:24.360273 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.360251    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 21:32:24.360515 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.360497    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 21:32:24.504906 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.504866    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-node-exporter-tls\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.505105 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.504917    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-node-exporter-wtmp\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.505105 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.504945    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsw4n\" (UniqueName: \"kubernetes.io/projected/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-kube-api-access-bsw4n\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.505105 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.505003    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.505105 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.505034    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.505105 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.505081    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-node-exporter-textfile\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.505454 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.505110    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-metrics-client-ca\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.505454 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.505154    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-sys\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.505454 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.505189    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-root\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.606192 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.606099    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.606192 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.606154    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.606192 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.606191    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-node-exporter-textfile\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.606535 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.606239    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-metrics-client-ca\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.606535 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.606271    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-sys\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.606535 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.606296    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-root\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.607014 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.606769    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-sys\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.607014 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.606876    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.607181 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.606981    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-root\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.607181 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.607073    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-metrics-client-ca\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.607181 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.606987    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-node-exporter-textfile\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.608189 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.607822    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-node-exporter-tls\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.608189 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.607855    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-node-exporter-wtmp\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.608189 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.607878    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsw4n\" (UniqueName: \"kubernetes.io/projected/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-kube-api-access-bsw4n\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.608189 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:32:24.607990    2544 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 21:32:24.608189 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.608014    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-node-exporter-wtmp\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.608189 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:32:24.608070    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-node-exporter-tls podName:c88f0aa7-fcd5-41b5-90ef-544e954dd7d6 nodeName:}" failed. No retries permitted until 2026-09-11 21:32:25.108048351 +0000 UTC m=+108.077465163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-node-exporter-tls") pod "node-exporter-lttfq" (UID: "c88f0aa7-fcd5-41b5-90ef-544e954dd7d6") : secret "node-exporter-tls" not found
Sep 11 21:32:24.609759 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.609718    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:24.618244 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:24.618139    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsw4n\" (UniqueName: \"kubernetes.io/projected/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-kube-api-access-bsw4n\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:25.075045 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:25.075005    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk" event={"ID":"7389ecba-055b-4054-8198-a927e4aca379","Type":"ContainerStarted","Data":"1e285859e8ed3bb7bb6cf789befe60a8139b5886be9b98e8776c6f63c5702bc3"}
Sep 11 21:32:25.075252 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:25.075188    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk"
Sep 11 21:32:25.076694 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:25.076671    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk"
Sep 11 21:32:25.091301 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:25.091242    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bf8db-4jgnk" podStartSLOduration=1.8017772399999998 podStartE2EDuration="17.091201405s" totalImagesPullingTime="15.289424165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:25.090858487 +0000 UTC m=+108.060275308" watchObservedRunningTime="2026-09-11 21:32:25.091201405 +0000 UTC m=+108.060618293"
Sep 11 21:32:25.111831 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:25.111790    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-node-exporter-tls\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:25.114139 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:25.114120    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c88f0aa7-fcd5-41b5-90ef-544e954dd7d6-node-exporter-tls\") pod \"node-exporter-lttfq\" (UID: \"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6\") " pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:25.269697 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:25.269661    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lttfq"
Sep 11 21:32:25.280351 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:32:25.280312    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88f0aa7_fcd5_41b5_90ef_544e954dd7d6.slice/crio-d245169a0c258300985c521d8fc8ac6650be737c3aaed7d8920b61528655468b WatchSource:0}: Error finding container d245169a0c258300985c521d8fc8ac6650be737c3aaed7d8920b61528655468b: Status 404 returned error can't find the container with id d245169a0c258300985c521d8fc8ac6650be737c3aaed7d8920b61528655468b
Sep 11 21:32:26.080235 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:26.080183    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lttfq" event={"ID":"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6","Type":"ContainerStarted","Data":"d245169a0c258300985c521d8fc8ac6650be737c3aaed7d8920b61528655468b"}
Sep 11 21:32:27.085132 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:27.085097    2544 generic.go:356] "Generic (PLEG): container finished" podID="c88f0aa7-fcd5-41b5-90ef-544e954dd7d6" containerID="4925c75c271bdbdbeb9cb3b38878598cfb241c99903b45d284c2c945bd59b99c" exitCode=0
Sep 11 21:32:27.085588 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:27.085173    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lttfq" event={"ID":"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6","Type":"ContainerDied","Data":"4925c75c271bdbdbeb9cb3b38878598cfb241c99903b45d284c2c945bd59b99c"}
Sep 11 21:32:28.050210 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:28.050181    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pnzdj"
Sep 11 21:32:28.090831 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:28.090795    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lttfq" event={"ID":"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6","Type":"ContainerStarted","Data":"4d0b92fcdf88dc113afc740fb5ef8478feafd62c753db68a5f3afed1b05665a7"}
Sep 11 21:32:28.091288 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:28.090850    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lttfq" event={"ID":"c88f0aa7-fcd5-41b5-90ef-544e954dd7d6","Type":"ContainerStarted","Data":"ac6228ceb8d57efdd657d209a8c84af59e51086d46efd6a681f933afabcf4018"}
Sep 11 21:32:28.111838 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:28.111784    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lttfq" podStartSLOduration=3.06092804 podStartE2EDuration="4.111765473s" totalImagesPullingTime="988.506279ms" totalInitContainerRuntime="62.331154ms" isStatefulPod=true podCreationTimestamp="2026-09-11 21:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:28.111066186 +0000 UTC m=+111.080483007" watchObservedRunningTime="2026-09-11 21:32:28.111765473 +0000 UTC m=+111.081182292"
Sep 11 21:32:47.279807 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:47.279756    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs\") pod \"network-metrics-daemon-k94pz\" (UID: \"047e6301-db2b-4c57-86ad-20ca80ff019a\") " pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:32:47.282018 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:47.281995    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047e6301-db2b-4c57-86ad-20ca80ff019a-metrics-certs\") pod \"network-metrics-daemon-k94pz\" (UID: \"047e6301-db2b-4c57-86ad-20ca80ff019a\") " pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:32:47.405958 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:47.405928    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r57v4\""
Sep 11 21:32:47.414262 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:47.414209    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k94pz"
Sep 11 21:32:47.744576 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:47.744539    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k94pz"]
Sep 11 21:32:47.747345 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:32:47.747314    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047e6301_db2b_4c57_86ad_20ca80ff019a.slice/crio-300df077b0923aee39e9b3387795a6eb21f7f831bbbb14b92de12f67e8ad5bca WatchSource:0}: Error finding container 300df077b0923aee39e9b3387795a6eb21f7f831bbbb14b92de12f67e8ad5bca: Status 404 returned error can't find the container with id 300df077b0923aee39e9b3387795a6eb21f7f831bbbb14b92de12f67e8ad5bca
Sep 11 21:32:48.160771 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:48.160677    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k94pz" event={"ID":"047e6301-db2b-4c57-86ad-20ca80ff019a","Type":"ContainerStarted","Data":"300df077b0923aee39e9b3387795a6eb21f7f831bbbb14b92de12f67e8ad5bca"}
Sep 11 21:32:49.166285 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:49.166244    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k94pz" event={"ID":"047e6301-db2b-4c57-86ad-20ca80ff019a","Type":"ContainerStarted","Data":"684c068de4df5a5f5cbcdda09a79066e1c142629b6a1126c2fe5fb170bf87f18"}
Sep 11 21:32:50.171188 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:50.171151    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k94pz" event={"ID":"047e6301-db2b-4c57-86ad-20ca80ff019a","Type":"ContainerStarted","Data":"e6d090b4b1a3496591dea0ba57b2aa399778bfc25a54c1fb37c97921429e11ef"}
Sep 11 21:32:50.188807 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:32:50.188754    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-k94pz" podStartSLOduration=131.992707377 podStartE2EDuration="2m13.188736714s" totalImagesPullingTime="1.196029337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:50.187196423 +0000 UTC m=+133.156613242" watchObservedRunningTime="2026-09-11 21:32:50.188736714 +0000 UTC m=+133.158153535"
Sep 11 21:35:37.463259 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:35:37.463209    2544 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 21:36:03.083608 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:03.080859    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"]
Sep 11 21:36:03.085678 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:03.085655    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:03.089154 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:36:03.089126    2544 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-142-211.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-211.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" type="*v1.Secret"
Sep 11 21:36:03.089154 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:36:03.089129    2544 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-142-211.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-211.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 11 21:36:03.089354 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:36:03.089129    2544 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-service-cert\" is forbidden: User \"system:node:ip-10-0-142-211.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-211.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret"
Sep 11 21:36:03.089623 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:36:03.089597    2544 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-142-211.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-211.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 11 21:36:03.090424 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:36:03.090401    2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"opendatahub-operator-controller-manager-595996979b-k85d5\" is forbidden: User \"system:node:ip-10-0-142-211.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-211.ec2.internal' and this object" podUID="f22c40a2-230b-460a-874f-3f1d71d7a491" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:03.090424 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:36:03.090402    2544 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-dockercfg-vmt8t\" is forbidden: User \"system:node:ip-10-0-142-211.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-211.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vmt8t\"" type="*v1.Secret"
Sep 11 21:36:03.110098 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:03.110077    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"]
Sep 11 21:36:03.197949 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:03.197915    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh2mk\" (UniqueName: \"kubernetes.io/projected/f22c40a2-230b-460a-874f-3f1d71d7a491-kube-api-access-hh2mk\") pod \"opendatahub-operator-controller-manager-595996979b-k85d5\" (UID: \"f22c40a2-230b-460a-874f-3f1d71d7a491\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:03.198108 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:03.197957    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f22c40a2-230b-460a-874f-3f1d71d7a491-apiservice-cert\") pod \"opendatahub-operator-controller-manager-595996979b-k85d5\" (UID: \"f22c40a2-230b-460a-874f-3f1d71d7a491\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:03.198108 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:03.198015    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f22c40a2-230b-460a-874f-3f1d71d7a491-tmp\") pod \"opendatahub-operator-controller-manager-595996979b-k85d5\" (UID: \"f22c40a2-230b-460a-874f-3f1d71d7a491\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:03.198108 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:03.198065    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f22c40a2-230b-460a-874f-3f1d71d7a491-webhook-cert\") pod \"opendatahub-operator-controller-manager-595996979b-k85d5\" (UID: \"f22c40a2-230b-460a-874f-3f1d71d7a491\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:03.198108 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:03.198096    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f22c40a2-230b-460a-874f-3f1d71d7a491-manifests\") pod \"opendatahub-operator-controller-manager-595996979b-k85d5\" (UID: \"f22c40a2-230b-460a-874f-3f1d71d7a491\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:03.299450 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:03.299418    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f22c40a2-230b-460a-874f-3f1d71d7a491-tmp\") pod \"opendatahub-operator-controller-manager-595996979b-k85d5\" (UID: \"f22c40a2-230b-460a-874f-3f1d71d7a491\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:03.299450 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:03.299455    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f22c40a2-230b-460a-874f-3f1d71d7a491-webhook-cert\") pod \"opendatahub-operator-controller-manager-595996979b-k85d5\" (UID: \"f22c40a2-230b-460a-874f-3f1d71d7a491\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:03.299673 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:03.299474    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f22c40a2-230b-460a-874f-3f1d71d7a491-manifests\") pod \"opendatahub-operator-controller-manager-595996979b-k85d5\" (UID: \"f22c40a2-230b-460a-874f-3f1d71d7a491\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:03.299673 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:03.299507    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh2mk\" (UniqueName: \"kubernetes.io/projected/f22c40a2-230b-460a-874f-3f1d71d7a491-kube-api-access-hh2mk\") pod \"opendatahub-operator-controller-manager-595996979b-k85d5\" (UID: \"f22c40a2-230b-460a-874f-3f1d71d7a491\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:03.299673 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:03.299539    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f22c40a2-230b-460a-874f-3f1d71d7a491-apiservice-cert\") pod \"opendatahub-operator-controller-manager-595996979b-k85d5\" (UID: \"f22c40a2-230b-460a-874f-3f1d71d7a491\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:03.299906 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:03.299883    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f22c40a2-230b-460a-874f-3f1d71d7a491-manifests\") pod \"opendatahub-operator-controller-manager-595996979b-k85d5\" (UID: \"f22c40a2-230b-460a-874f-3f1d71d7a491\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:03.301558 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:03.301539    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f22c40a2-230b-460a-874f-3f1d71d7a491-tmp\") pod \"opendatahub-operator-controller-manager-595996979b-k85d5\" (UID: \"f22c40a2-230b-460a-874f-3f1d71d7a491\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:03.995195 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:03.995163    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vmt8t\""
Sep 11 21:36:03.998324 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:03.998302    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 21:36:04.002530 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:04.002511    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f22c40a2-230b-460a-874f-3f1d71d7a491-webhook-cert\") pod \"opendatahub-operator-controller-manager-595996979b-k85d5\" (UID: \"f22c40a2-230b-460a-874f-3f1d71d7a491\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:04.002674 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:04.002654    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f22c40a2-230b-460a-874f-3f1d71d7a491-apiservice-cert\") pod \"opendatahub-operator-controller-manager-595996979b-k85d5\" (UID: \"f22c40a2-230b-460a-874f-3f1d71d7a491\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:04.030270 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:04.030249    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 21:36:04.291747 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:04.291672    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 21:36:04.300941 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:04.300909    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh2mk\" (UniqueName: \"kubernetes.io/projected/f22c40a2-230b-460a-874f-3f1d71d7a491-kube-api-access-hh2mk\") pod \"opendatahub-operator-controller-manager-595996979b-k85d5\" (UID: \"f22c40a2-230b-460a-874f-3f1d71d7a491\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:04.559966 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:04.559889    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 21:36:04.595691 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:04.595663    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:04.867048 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:04.867025    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"]
Sep 11 21:36:04.868720 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:36:04.868694    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22c40a2_230b_460a_874f_3f1d71d7a491.slice/crio-016c1d964f2c8959fe8be6cbc7600e78f959c242c4561f34ecbb0b9a2d971aab WatchSource:0}: Error finding container 016c1d964f2c8959fe8be6cbc7600e78f959c242c4561f34ecbb0b9a2d971aab: Status 404 returned error can't find the container with id 016c1d964f2c8959fe8be6cbc7600e78f959c242c4561f34ecbb0b9a2d971aab
Sep 11 21:36:04.870581 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:04.870565    2544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:36:05.850394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:05.850345    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5" event={"ID":"f22c40a2-230b-460a-874f-3f1d71d7a491","Type":"ContainerStarted","Data":"016c1d964f2c8959fe8be6cbc7600e78f959c242c4561f34ecbb0b9a2d971aab"}
Sep 11 21:36:07.859143 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:07.859065    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5" event={"ID":"f22c40a2-230b-460a-874f-3f1d71d7a491","Type":"ContainerStarted","Data":"211c88321561526a48d433e7bdabd270b6c388300da24e66e9e62249abb0579f"}
Sep 11 21:36:08.863817 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:08.863732    2544 generic.go:356] "Generic (PLEG): container finished" podID="f22c40a2-230b-460a-874f-3f1d71d7a491" containerID="211c88321561526a48d433e7bdabd270b6c388300da24e66e9e62249abb0579f" exitCode=0
Sep 11 21:36:08.863817 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:08.863808    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5" event={"ID":"f22c40a2-230b-460a-874f-3f1d71d7a491","Type":"ContainerDied","Data":"211c88321561526a48d433e7bdabd270b6c388300da24e66e9e62249abb0579f"}
Sep 11 21:36:09.869879 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:09.869846    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5" event={"ID":"f22c40a2-230b-460a-874f-3f1d71d7a491","Type":"ContainerStarted","Data":"e0ba5e58e4488a6a5c225c17679763e111c37cd411e4234cca340f509240e6a0"}
Sep 11 21:36:09.870278 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:09.869964    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:09.891455 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:09.891412    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5" podStartSLOduration=2.910743529 podStartE2EDuration="6.891396988s" totalImagesPullingTime="2.998398296s" totalInitContainerRuntime="982.255163ms" isStatefulPod=false podCreationTimestamp="2026-09-11 21:36:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:36:09.890552319 +0000 UTC m=+332.859969138" watchObservedRunningTime="2026-09-11 21:36:09.891396988 +0000 UTC m=+332.860813807"
Sep 11 21:36:20.876163 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:20.876133    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-k85d5"
Sep 11 21:36:22.948732 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:22.948701    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-57ddc9f5f6-4tg7c"]
Sep 11 21:36:22.957415 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:22.957392    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57ddc9f5f6-4tg7c"
Sep 11 21:36:22.960208 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:22.960188    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 21:36:22.960313 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:22.960288    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 21:36:22.960394 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:22.960371    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-g89m4\""
Sep 11 21:36:22.969797 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:22.969777    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57ddc9f5f6-4tg7c"]
Sep 11 21:36:23.036731 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:23.036704    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d4be6af-1f09-48dd-baac-d08a216c34e2-tmp\") pod \"kube-auth-proxy-57ddc9f5f6-4tg7c\" (UID: \"0d4be6af-1f09-48dd-baac-d08a216c34e2\") " pod="openshift-ingress/kube-auth-proxy-57ddc9f5f6-4tg7c"
Sep 11 21:36:23.036892 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:23.036741    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d9hj\" (UniqueName: \"kubernetes.io/projected/0d4be6af-1f09-48dd-baac-d08a216c34e2-kube-api-access-4d9hj\") pod \"kube-auth-proxy-57ddc9f5f6-4tg7c\" (UID: \"0d4be6af-1f09-48dd-baac-d08a216c34e2\") " pod="openshift-ingress/kube-auth-proxy-57ddc9f5f6-4tg7c"
Sep 11 21:36:23.036892 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:23.036807    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d4be6af-1f09-48dd-baac-d08a216c34e2-tls-certs\") pod \"kube-auth-proxy-57ddc9f5f6-4tg7c\" (UID: \"0d4be6af-1f09-48dd-baac-d08a216c34e2\") " pod="openshift-ingress/kube-auth-proxy-57ddc9f5f6-4tg7c"
Sep 11 21:36:23.137474 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:23.137441    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d4be6af-1f09-48dd-baac-d08a216c34e2-tls-certs\") pod \"kube-auth-proxy-57ddc9f5f6-4tg7c\" (UID: \"0d4be6af-1f09-48dd-baac-d08a216c34e2\") " pod="openshift-ingress/kube-auth-proxy-57ddc9f5f6-4tg7c"
Sep 11 21:36:23.137637 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:23.137507    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d4be6af-1f09-48dd-baac-d08a216c34e2-tmp\") pod \"kube-auth-proxy-57ddc9f5f6-4tg7c\" (UID: \"0d4be6af-1f09-48dd-baac-d08a216c34e2\") " pod="openshift-ingress/kube-auth-proxy-57ddc9f5f6-4tg7c"
Sep 11 21:36:23.137637 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:23.137534    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d9hj\" (UniqueName: \"kubernetes.io/projected/0d4be6af-1f09-48dd-baac-d08a216c34e2-kube-api-access-4d9hj\") pod \"kube-auth-proxy-57ddc9f5f6-4tg7c\" (UID: \"0d4be6af-1f09-48dd-baac-d08a216c34e2\") " pod="openshift-ingress/kube-auth-proxy-57ddc9f5f6-4tg7c"
Sep 11 21:36:23.139801 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:23.139778    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d4be6af-1f09-48dd-baac-d08a216c34e2-tmp\") pod \"kube-auth-proxy-57ddc9f5f6-4tg7c\" (UID: \"0d4be6af-1f09-48dd-baac-d08a216c34e2\") " pod="openshift-ingress/kube-auth-proxy-57ddc9f5f6-4tg7c"
Sep 11 21:36:23.139942 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:23.139924    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d4be6af-1f09-48dd-baac-d08a216c34e2-tls-certs\") pod \"kube-auth-proxy-57ddc9f5f6-4tg7c\" (UID: \"0d4be6af-1f09-48dd-baac-d08a216c34e2\") " pod="openshift-ingress/kube-auth-proxy-57ddc9f5f6-4tg7c"
Sep 11 21:36:23.146238 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:23.146202    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d9hj\" (UniqueName: \"kubernetes.io/projected/0d4be6af-1f09-48dd-baac-d08a216c34e2-kube-api-access-4d9hj\") pod \"kube-auth-proxy-57ddc9f5f6-4tg7c\" (UID: \"0d4be6af-1f09-48dd-baac-d08a216c34e2\") " pod="openshift-ingress/kube-auth-proxy-57ddc9f5f6-4tg7c"
Sep 11 21:36:23.267965 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:23.267922    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57ddc9f5f6-4tg7c"
Sep 11 21:36:23.542318 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:36:23.542287    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4be6af_1f09_48dd_baac_d08a216c34e2.slice/crio-1f0b472066202acb9e626ea441c8df96b3def6b58c366c1ea257e6c51a43d5e1 WatchSource:0}: Error finding container 1f0b472066202acb9e626ea441c8df96b3def6b58c366c1ea257e6c51a43d5e1: Status 404 returned error can't find the container with id 1f0b472066202acb9e626ea441c8df96b3def6b58c366c1ea257e6c51a43d5e1
Sep 11 21:36:23.551685 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:23.551659    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57ddc9f5f6-4tg7c"]
Sep 11 21:36:23.920726 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:23.920635    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57ddc9f5f6-4tg7c" event={"ID":"0d4be6af-1f09-48dd-baac-d08a216c34e2","Type":"ContainerStarted","Data":"1f0b472066202acb9e626ea441c8df96b3def6b58c366c1ea257e6c51a43d5e1"}
Sep 11 21:36:26.941112 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:26.941072    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57ddc9f5f6-4tg7c" event={"ID":"0d4be6af-1f09-48dd-baac-d08a216c34e2","Type":"ContainerStarted","Data":"aed34658d3a907f4721e56922169ba47670075d1c60072f64544e4bc6d5458c2"}
Sep 11 21:36:26.957790 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:26.957746    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-57ddc9f5f6-4tg7c" podStartSLOduration=2.330188072 podStartE2EDuration="4.957731662s" totalImagesPullingTime="2.62754359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:36:26.956445853 +0000 UTC m=+349.925862672" watchObservedRunningTime="2026-09-11 21:36:26.957731662 +0000 UTC m=+349.927148480"
Sep 11 21:36:30.176456 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.176371    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"]
Sep 11 21:36:30.179330 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.179314    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.181819 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.181796    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-r92dm\""
Sep 11 21:36:30.182212 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.182193    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 21:36:30.195127 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.195100    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"]
Sep 11 21:36:30.278632 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.278606    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"]
Sep 11 21:36:30.281506 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.281492    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.290235 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.290193    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.290312 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.290247    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.290312 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.290288    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.290383 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.290330    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.290383 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.290348    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.290383 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.290363    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx8s5\" (UniqueName: \"kubernetes.io/projected/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-kube-api-access-jx8s5\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.290516 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.290384    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.290516 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.290414    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.290516 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.290487    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.294142 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.294121    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"]
Sep 11 21:36:30.391583 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.391548    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.391751 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.391589    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.391751 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.391619    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.391751 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.391657    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vshbh\" (UniqueName: \"kubernetes.io/projected/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-kube-api-access-vshbh\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.391751 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.391734    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.391938 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.391758    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.391938 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.391778    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.391938 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.391819    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.391938 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.391836    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.391938 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.391850    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx8s5\" (UniqueName: \"kubernetes.io/projected/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-kube-api-access-jx8s5\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.391938 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.391871    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.391938 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.391886    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.391938 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.391901    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.391938 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.391930    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.392355 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.391954    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.392355 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.391976    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.392355 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.392001    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.392355 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.392032    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.392355 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.392265    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.392554 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.392537    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.392623 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.392601    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.392806 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.392787    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.393043 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.393021    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.394140 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.394113    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.394363 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.394341    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.403608 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.403590    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx8s5\" (UniqueName: \"kubernetes.io/projected/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-kube-api-access-jx8s5\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.403706 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.403686    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcn99nw\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.489427 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.489403    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:30.492645 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.492625    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.492730 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.492655    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.492730 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.492693    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.492730 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.492722    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.492882 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.492754    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.492882 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.492779    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vshbh\" (UniqueName: \"kubernetes.io/projected/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-kube-api-access-vshbh\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.493027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.493001    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.493233 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.493039    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.493233 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.493063    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.493233 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.493151    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.493403 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.493344    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.493775 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.493462    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.493775 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.493509    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.493945 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.493782    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.494956 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.494931    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.495508 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.495486    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.502289 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.502266    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vshbh\" (UniqueName: \"kubernetes.io/projected/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-kube-api-access-vshbh\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.502934 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.502914    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7429fa7-e597-47cd-9ef9-ceb95ded04dc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs\" (UID: \"c7429fa7-e597-47cd-9ef9-ceb95ded04dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.591198 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.591167    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:30.769739 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.769713    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"]
Sep 11 21:36:30.771839 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:36:30.771813    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb3aa1a_38e8_4706_9962_dd3fa9c30e89.slice/crio-55399a82afd48bfabfad3e420aae51e1c9e59105a86a5f5342295ebb1a26e15b WatchSource:0}: Error finding container 55399a82afd48bfabfad3e420aae51e1c9e59105a86a5f5342295ebb1a26e15b: Status 404 returned error can't find the container with id 55399a82afd48bfabfad3e420aae51e1c9e59105a86a5f5342295ebb1a26e15b
Sep 11 21:36:30.863265 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.863241    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"]
Sep 11 21:36:30.865107 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:36:30.865086    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7429fa7_e597_47cd_9ef9_ceb95ded04dc.slice/crio-c4ae3cd5ccdfea0e47c905b3156910c637edc07ee01c2dd5db687dfe9c2d352c WatchSource:0}: Error finding container c4ae3cd5ccdfea0e47c905b3156910c637edc07ee01c2dd5db687dfe9c2d352c: Status 404 returned error can't find the container with id c4ae3cd5ccdfea0e47c905b3156910c637edc07ee01c2dd5db687dfe9c2d352c
Sep 11 21:36:30.955480 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.955450    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs" event={"ID":"c7429fa7-e597-47cd-9ef9-ceb95ded04dc","Type":"ContainerStarted","Data":"c4ae3cd5ccdfea0e47c905b3156910c637edc07ee01c2dd5db687dfe9c2d352c"}
Sep 11 21:36:30.956618 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:30.956596    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw" event={"ID":"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89","Type":"ContainerStarted","Data":"55399a82afd48bfabfad3e420aae51e1c9e59105a86a5f5342295ebb1a26e15b"}
Sep 11 21:36:33.623700 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:33.623647    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:36:33.624000 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:33.623741    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:36:33.624000 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:33.623792    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:36:33.629743 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:33.629708    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:36:33.629861 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:33.629794    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:36:33.629861 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:33.629838    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:36:33.971839 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:33.971762    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw" event={"ID":"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89","Type":"ContainerStarted","Data":"f7639e1ee8cbe827188cfba6d83149423181cffd2820432204f7d436c216a7d2"}
Sep 11 21:36:33.973444 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:33.973416    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs" event={"ID":"c7429fa7-e597-47cd-9ef9-ceb95ded04dc","Type":"ContainerStarted","Data":"3ae66f823713c99a5619b780fd4a882b53d1519c575bbd6ab82bf95d41084019"}
Sep 11 21:36:33.999857 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:33.999805    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw" podStartSLOduration=1.150245623 podStartE2EDuration="3.999790862s" totalImagesPullingTime="2.849545239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:36:33.998473833 +0000 UTC m=+356.967890652" watchObservedRunningTime="2026-09-11 21:36:33.999790862 +0000 UTC m=+356.969207677"
Sep 11 21:36:34.020182 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:34.020134    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs" podStartSLOduration=1.257763931 podStartE2EDuration="4.020115575s" totalImagesPullingTime="2.762351644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:36:34.019329529 +0000 UTC m=+356.988746347" watchObservedRunningTime="2026-09-11 21:36:34.020115575 +0000 UTC m=+356.989532394"
Sep 11 21:36:34.489746 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:34.489712    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:34.490829 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:34.490803    2544 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" start-of-body=
Sep 11 21:36:34.490925 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:34.490850    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw" podUID="5eb3aa1a-38e8-4706-9962-dd3fa9c30e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused"
Sep 11 21:36:34.592194 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:34.592165    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:34.596434 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:34.596413    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:34.977630 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:34.977594    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:34.978424 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:34.978405    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs"
Sep 11 21:36:35.027584 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:35.027552    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"]
Sep 11 21:36:35.490703 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:35.490667    2544 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" start-of-body=
Sep 11 21:36:35.490893 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:35.490736    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw" podUID="5eb3aa1a-38e8-4706-9962-dd3fa9c30e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused"
Sep 11 21:36:36.490185 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:36.490141    2544 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" start-of-body=
Sep 11 21:36:36.490591 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:36.490213    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw" podUID="5eb3aa1a-38e8-4706-9962-dd3fa9c30e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused"
Sep 11 21:36:36.985993 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:36.985956    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw" podUID="5eb3aa1a-38e8-4706-9962-dd3fa9c30e89" containerName="istio-proxy" containerID="cri-o://f7639e1ee8cbe827188cfba6d83149423181cffd2820432204f7d436c216a7d2" gracePeriod=30
Sep 11 21:36:40.593848 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.593814    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"]
Sep 11 21:36:40.598645 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.598628    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"
Sep 11 21:36:40.601338 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.601290    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 21:36:40.602412 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.602392    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 21:36:40.602412 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.602409    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 21:36:40.602989 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.602971    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 21:36:40.603056 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.602990    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:36:40.603130 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.603055    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4mq2d\""
Sep 11 21:36:40.607894 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.607872    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"]
Sep 11 21:36:40.673945 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.673909    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6188bae4-f1e3-4668-9b1f-1ddfab718dc8-cert\") pod \"lws-controller-manager-74f4c6fbc8-qhtdk\" (UID: \"6188bae4-f1e3-4668-9b1f-1ddfab718dc8\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"
Sep 11 21:36:40.674095 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.673962    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6188bae4-f1e3-4668-9b1f-1ddfab718dc8-manager-config\") pod \"lws-controller-manager-74f4c6fbc8-qhtdk\" (UID: \"6188bae4-f1e3-4668-9b1f-1ddfab718dc8\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"
Sep 11 21:36:40.674095 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.673989    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6188bae4-f1e3-4668-9b1f-1ddfab718dc8-metrics-cert\") pod \"lws-controller-manager-74f4c6fbc8-qhtdk\" (UID: \"6188bae4-f1e3-4668-9b1f-1ddfab718dc8\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"
Sep 11 21:36:40.674095 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.674006    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmft\" (UniqueName: \"kubernetes.io/projected/6188bae4-f1e3-4668-9b1f-1ddfab718dc8-kube-api-access-6hmft\") pod \"lws-controller-manager-74f4c6fbc8-qhtdk\" (UID: \"6188bae4-f1e3-4668-9b1f-1ddfab718dc8\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"
Sep 11 21:36:40.774675 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.774636    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6188bae4-f1e3-4668-9b1f-1ddfab718dc8-cert\") pod \"lws-controller-manager-74f4c6fbc8-qhtdk\" (UID: \"6188bae4-f1e3-4668-9b1f-1ddfab718dc8\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"
Sep 11 21:36:40.774832 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.774691    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6188bae4-f1e3-4668-9b1f-1ddfab718dc8-manager-config\") pod \"lws-controller-manager-74f4c6fbc8-qhtdk\" (UID: \"6188bae4-f1e3-4668-9b1f-1ddfab718dc8\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"
Sep 11 21:36:40.774832 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.774715    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6188bae4-f1e3-4668-9b1f-1ddfab718dc8-metrics-cert\") pod \"lws-controller-manager-74f4c6fbc8-qhtdk\" (UID: \"6188bae4-f1e3-4668-9b1f-1ddfab718dc8\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"
Sep 11 21:36:40.774832 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.774732    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmft\" (UniqueName: \"kubernetes.io/projected/6188bae4-f1e3-4668-9b1f-1ddfab718dc8-kube-api-access-6hmft\") pod \"lws-controller-manager-74f4c6fbc8-qhtdk\" (UID: \"6188bae4-f1e3-4668-9b1f-1ddfab718dc8\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"
Sep 11 21:36:40.775375 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.775354    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6188bae4-f1e3-4668-9b1f-1ddfab718dc8-manager-config\") pod \"lws-controller-manager-74f4c6fbc8-qhtdk\" (UID: \"6188bae4-f1e3-4668-9b1f-1ddfab718dc8\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"
Sep 11 21:36:40.777054 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.777037    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6188bae4-f1e3-4668-9b1f-1ddfab718dc8-metrics-cert\") pod \"lws-controller-manager-74f4c6fbc8-qhtdk\" (UID: \"6188bae4-f1e3-4668-9b1f-1ddfab718dc8\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"
Sep 11 21:36:40.777109 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.777075    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6188bae4-f1e3-4668-9b1f-1ddfab718dc8-cert\") pod \"lws-controller-manager-74f4c6fbc8-qhtdk\" (UID: \"6188bae4-f1e3-4668-9b1f-1ddfab718dc8\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"
Sep 11 21:36:40.786047 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.786020    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmft\" (UniqueName: \"kubernetes.io/projected/6188bae4-f1e3-4668-9b1f-1ddfab718dc8-kube-api-access-6hmft\") pod \"lws-controller-manager-74f4c6fbc8-qhtdk\" (UID: \"6188bae4-f1e3-4668-9b1f-1ddfab718dc8\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"
Sep 11 21:36:40.907900 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:40.907815    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"
Sep 11 21:36:41.179054 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:41.179029    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"]
Sep 11 21:36:41.181287 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:36:41.181244    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6188bae4_f1e3_4668_9b1f_1ddfab718dc8.slice/crio-ca00ea39664ebb7624bedddde547949d53e8aa81a9b8ce7b75e288215ce7fd76 WatchSource:0}: Error finding container ca00ea39664ebb7624bedddde547949d53e8aa81a9b8ce7b75e288215ce7fd76: Status 404 returned error can't find the container with id ca00ea39664ebb7624bedddde547949d53e8aa81a9b8ce7b75e288215ce7fd76
Sep 11 21:36:42.007310 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.007272    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk" event={"ID":"6188bae4-f1e3-4668-9b1f-1ddfab718dc8","Type":"ContainerStarted","Data":"ca00ea39664ebb7624bedddde547949d53e8aa81a9b8ce7b75e288215ce7fd76"}
Sep 11 21:36:42.263637 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.263570    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:42.388109 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.388078    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-data\") pod \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") "
Sep 11 21:36:42.388291 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.388123    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-envoy\") pod \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") "
Sep 11 21:36:42.388291 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.388161    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-credential-socket\") pod \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") "
Sep 11 21:36:42.388291 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.388187    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-podinfo\") pod \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") "
Sep 11 21:36:42.388291 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.388211    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-workload-certs\") pod \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") "
Sep 11 21:36:42.388291 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.388249    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istiod-ca-cert\") pod \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") "
Sep 11 21:36:42.388291 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.388276    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-token\") pod \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") "
Sep 11 21:36:42.388570 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.388309    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-workload-socket\") pod \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") "
Sep 11 21:36:42.388570 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.388349    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx8s5\" (UniqueName: \"kubernetes.io/projected/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-kube-api-access-jx8s5\") pod \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\" (UID: \"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89\") "
Sep 11 21:36:42.390520 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.390476    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-data" (OuterVolumeSpecName: "istio-data") pod "5eb3aa1a-38e8-4706-9962-dd3fa9c30e89" (UID: "5eb3aa1a-38e8-4706-9962-dd3fa9c30e89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:36:42.390783 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.390694    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5eb3aa1a-38e8-4706-9962-dd3fa9c30e89" (UID: "5eb3aa1a-38e8-4706-9962-dd3fa9c30e89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:36:42.390783 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.390679    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5eb3aa1a-38e8-4706-9962-dd3fa9c30e89" (UID: "5eb3aa1a-38e8-4706-9962-dd3fa9c30e89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:36:42.390996 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.390953    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5eb3aa1a-38e8-4706-9962-dd3fa9c30e89" (UID: "5eb3aa1a-38e8-4706-9962-dd3fa9c30e89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:36:42.392537 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.392508    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5eb3aa1a-38e8-4706-9962-dd3fa9c30e89" (UID: "5eb3aa1a-38e8-4706-9962-dd3fa9c30e89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:36:42.392712 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.392596    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5eb3aa1a-38e8-4706-9962-dd3fa9c30e89" (UID: "5eb3aa1a-38e8-4706-9962-dd3fa9c30e89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:36:42.393571 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.393455    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5eb3aa1a-38e8-4706-9962-dd3fa9c30e89" (UID: "5eb3aa1a-38e8-4706-9962-dd3fa9c30e89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:36:42.395484 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.395442    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-token" (OuterVolumeSpecName: "istio-token") pod "5eb3aa1a-38e8-4706-9962-dd3fa9c30e89" (UID: "5eb3aa1a-38e8-4706-9962-dd3fa9c30e89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:36:42.395833 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.395811    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-kube-api-access-jx8s5" (OuterVolumeSpecName: "kube-api-access-jx8s5") pod "5eb3aa1a-38e8-4706-9962-dd3fa9c30e89" (UID: "5eb3aa1a-38e8-4706-9962-dd3fa9c30e89"). InnerVolumeSpecName "kube-api-access-jx8s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:36:42.489157 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.489123    2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-data\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:36:42.489157 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.489156    2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-envoy\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:36:42.489402 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.489169    2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-credential-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:36:42.489402 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.489182    2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-podinfo\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:36:42.489402 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.489191    2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-workload-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:36:42.489402 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.489199    2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istiod-ca-cert\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:36:42.489402 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.489207    2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-istio-token\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:36:42.489402 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.489214    2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-workload-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:36:42.489402 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:42.489242    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx8s5\" (UniqueName: \"kubernetes.io/projected/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89-kube-api-access-jx8s5\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:36:43.012855 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:43.012820    2544 generic.go:356] "Generic (PLEG): container finished" podID="5eb3aa1a-38e8-4706-9962-dd3fa9c30e89" containerID="f7639e1ee8cbe827188cfba6d83149423181cffd2820432204f7d436c216a7d2" exitCode=0
Sep 11 21:36:43.013359 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:43.012880    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"
Sep 11 21:36:43.013359 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:43.012881    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw" event={"ID":"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89","Type":"ContainerDied","Data":"f7639e1ee8cbe827188cfba6d83149423181cffd2820432204f7d436c216a7d2"}
Sep 11 21:36:43.013359 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:43.013010    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw" event={"ID":"5eb3aa1a-38e8-4706-9962-dd3fa9c30e89","Type":"ContainerDied","Data":"55399a82afd48bfabfad3e420aae51e1c9e59105a86a5f5342295ebb1a26e15b"}
Sep 11 21:36:43.013359 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:43.013039    2544 scope.go:117] "RemoveContainer" containerID="f7639e1ee8cbe827188cfba6d83149423181cffd2820432204f7d436c216a7d2"
Sep 11 21:36:43.050667 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:43.050581    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"]
Sep 11 21:36:43.054667 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:43.054635    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcn99nw"]
Sep 11 21:36:43.501437 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:43.501366    2544 scope.go:117] "RemoveContainer" containerID="f7639e1ee8cbe827188cfba6d83149423181cffd2820432204f7d436c216a7d2"
Sep 11 21:36:43.501734 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:36:43.501714    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7639e1ee8cbe827188cfba6d83149423181cffd2820432204f7d436c216a7d2\": container with ID starting with f7639e1ee8cbe827188cfba6d83149423181cffd2820432204f7d436c216a7d2 not found: ID does not exist" containerID="f7639e1ee8cbe827188cfba6d83149423181cffd2820432204f7d436c216a7d2"
Sep 11 21:36:43.501789 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:43.501741    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7639e1ee8cbe827188cfba6d83149423181cffd2820432204f7d436c216a7d2"} err="failed to get container status \"f7639e1ee8cbe827188cfba6d83149423181cffd2820432204f7d436c216a7d2\": rpc error: code = NotFound desc = could not find container \"f7639e1ee8cbe827188cfba6d83149423181cffd2820432204f7d436c216a7d2\": container with ID starting with f7639e1ee8cbe827188cfba6d83149423181cffd2820432204f7d436c216a7d2 not found: ID does not exist"
Sep 11 21:36:43.590686 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:43.590660    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eb3aa1a-38e8-4706-9962-dd3fa9c30e89" path="/var/lib/kubelet/pods/5eb3aa1a-38e8-4706-9962-dd3fa9c30e89/volumes"
Sep 11 21:36:44.018353 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:44.018313    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk" event={"ID":"6188bae4-f1e3-4668-9b1f-1ddfab718dc8","Type":"ContainerStarted","Data":"e087db1c09691d8103ad3245c045c036338329dca62d83cea61499b61bd053f7"}
Sep 11 21:36:44.018813 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:44.018496    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"
Sep 11 21:36:44.041863 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:44.041821    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk" podStartSLOduration=1.6658845169999998 podStartE2EDuration="4.041806706s" totalImagesPullingTime="2.375922189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:36:44.039300762 +0000 UTC m=+367.008717580" watchObservedRunningTime="2026-09-11 21:36:44.041806706 +0000 UTC m=+367.011223603"
Sep 11 21:36:55.026645 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:36:55.026610    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-qhtdk"
Sep 11 21:39:18.145077 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.145037    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lbm7j"]
Sep 11 21:39:18.150312 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.145941    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eb3aa1a-38e8-4706-9962-dd3fa9c30e89" containerName="istio-proxy"
Sep 11 21:39:18.150312 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.145970    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eb3aa1a-38e8-4706-9962-dd3fa9c30e89" containerName="istio-proxy"
Sep 11 21:39:18.150312 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.146119    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eb3aa1a-38e8-4706-9962-dd3fa9c30e89" containerName="istio-proxy"
Sep 11 21:39:18.151639 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.151619    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lbm7j"
Sep 11 21:39:18.154133 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.154109    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 21:39:18.154265 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.154162    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 21:39:18.154265 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.154190    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-p87jn\""
Sep 11 21:39:18.156789 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.156760    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lbm7j"]
Sep 11 21:39:18.331002 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.330958    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj97g\" (UniqueName: \"kubernetes.io/projected/0da27985-0a5d-4a58-829b-82ed49dd67cd-kube-api-access-wj97g\") pod \"authorino-7bcfd8f9cd-lbm7j\" (UID: \"0da27985-0a5d-4a58-829b-82ed49dd67cd\") " pod="kuadrant-system/authorino-7bcfd8f9cd-lbm7j"
Sep 11 21:39:18.379915 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.379879    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-j9kgc"]
Sep 11 21:39:18.383038 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.383018    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-j9kgc"
Sep 11 21:39:18.390635 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.390608    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-j9kgc"]
Sep 11 21:39:18.432077 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.431977    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj97g\" (UniqueName: \"kubernetes.io/projected/0da27985-0a5d-4a58-829b-82ed49dd67cd-kube-api-access-wj97g\") pod \"authorino-7bcfd8f9cd-lbm7j\" (UID: \"0da27985-0a5d-4a58-829b-82ed49dd67cd\") " pod="kuadrant-system/authorino-7bcfd8f9cd-lbm7j"
Sep 11 21:39:18.439917 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.439885    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj97g\" (UniqueName: \"kubernetes.io/projected/0da27985-0a5d-4a58-829b-82ed49dd67cd-kube-api-access-wj97g\") pod \"authorino-7bcfd8f9cd-lbm7j\" (UID: \"0da27985-0a5d-4a58-829b-82ed49dd67cd\") " pod="kuadrant-system/authorino-7bcfd8f9cd-lbm7j"
Sep 11 21:39:18.461554 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.461519    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lbm7j"
Sep 11 21:39:18.533098 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.533062    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zckfv\" (UniqueName: \"kubernetes.io/projected/26400880-a218-4304-8e8d-7b9875c66310-kube-api-access-zckfv\") pod \"authorino-59c974bd48-j9kgc\" (UID: \"26400880-a218-4304-8e8d-7b9875c66310\") " pod="kuadrant-system/authorino-59c974bd48-j9kgc"
Sep 11 21:39:18.634146 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.634103    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zckfv\" (UniqueName: \"kubernetes.io/projected/26400880-a218-4304-8e8d-7b9875c66310-kube-api-access-zckfv\") pod \"authorino-59c974bd48-j9kgc\" (UID: \"26400880-a218-4304-8e8d-7b9875c66310\") " pod="kuadrant-system/authorino-59c974bd48-j9kgc"
Sep 11 21:39:18.644341 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.644314    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zckfv\" (UniqueName: \"kubernetes.io/projected/26400880-a218-4304-8e8d-7b9875c66310-kube-api-access-zckfv\") pod \"authorino-59c974bd48-j9kgc\" (UID: \"26400880-a218-4304-8e8d-7b9875c66310\") " pod="kuadrant-system/authorino-59c974bd48-j9kgc"
Sep 11 21:39:18.692198 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.692113    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-j9kgc"
Sep 11 21:39:18.731961 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.731926    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lbm7j"]
Sep 11 21:39:18.735425 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:39:18.735395    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0da27985_0a5d_4a58_829b_82ed49dd67cd.slice/crio-ce51b869ac6786e0a89a84e82a719ca69f2c7362afbe347d5f97f62f695d34e7 WatchSource:0}: Error finding container ce51b869ac6786e0a89a84e82a719ca69f2c7362afbe347d5f97f62f695d34e7: Status 404 returned error can't find the container with id ce51b869ac6786e0a89a84e82a719ca69f2c7362afbe347d5f97f62f695d34e7
Sep 11 21:39:18.964543 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:18.964515    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-j9kgc"]
Sep 11 21:39:18.966857 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:39:18.966827    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26400880_a218_4304_8e8d_7b9875c66310.slice/crio-0671fab8f3b7a7323a116a3116eb3d6868e8a64f38170e364bf9002e9078ea16 WatchSource:0}: Error finding container 0671fab8f3b7a7323a116a3116eb3d6868e8a64f38170e364bf9002e9078ea16: Status 404 returned error can't find the container with id 0671fab8f3b7a7323a116a3116eb3d6868e8a64f38170e364bf9002e9078ea16
Sep 11 21:39:19.625466 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:19.625426    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lbm7j" event={"ID":"0da27985-0a5d-4a58-829b-82ed49dd67cd","Type":"ContainerStarted","Data":"ce51b869ac6786e0a89a84e82a719ca69f2c7362afbe347d5f97f62f695d34e7"}
Sep 11 21:39:19.626452 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:19.626429    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-j9kgc" event={"ID":"26400880-a218-4304-8e8d-7b9875c66310","Type":"ContainerStarted","Data":"0671fab8f3b7a7323a116a3116eb3d6868e8a64f38170e364bf9002e9078ea16"}
Sep 11 21:39:23.645156 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:23.645116    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lbm7j" event={"ID":"0da27985-0a5d-4a58-829b-82ed49dd67cd","Type":"ContainerStarted","Data":"6a66ad9eedfe6c973734c61c70f36a97a8d62eca521d991ebfc8a4ddd420f88b"}
Sep 11 21:39:23.646697 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:23.646669    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-j9kgc" event={"ID":"26400880-a218-4304-8e8d-7b9875c66310","Type":"ContainerStarted","Data":"286763dee525cf3898c509886418ca34d1858d31fcb6f336a69584575524af3e"}
Sep 11 21:39:23.660145 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:23.660093    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-lbm7j" podStartSLOduration=0.917531235 podStartE2EDuration="5.66007648s" totalImagesPullingTime="4.742545245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:39:23.658937385 +0000 UTC m=+526.628354217" watchObservedRunningTime="2026-09-11 21:39:23.66007648 +0000 UTC m=+526.629493299"
Sep 11 21:39:23.675588 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:23.675535    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-j9kgc" podStartSLOduration=1.154396297 podStartE2EDuration="5.675517359s" totalImagesPullingTime="4.521121062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:39:23.674034391 +0000 UTC m=+526.643451212" watchObservedRunningTime="2026-09-11 21:39:23.675517359 +0000 UTC m=+526.644934203"
Sep 11 21:39:23.702329 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:23.702291    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lbm7j"]
Sep 11 21:39:25.655046 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:25.655005    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-lbm7j" podUID="0da27985-0a5d-4a58-829b-82ed49dd67cd" containerName="authorino" containerID="cri-o://6a66ad9eedfe6c973734c61c70f36a97a8d62eca521d991ebfc8a4ddd420f88b" gracePeriod=30
Sep 11 21:39:25.896336 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:25.896310    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lbm7j"
Sep 11 21:39:25.998575 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:25.998524    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wj97g\" (UniqueName: \"kubernetes.io/projected/0da27985-0a5d-4a58-829b-82ed49dd67cd-kube-api-access-wj97g\") pod \"0da27985-0a5d-4a58-829b-82ed49dd67cd\" (UID: \"0da27985-0a5d-4a58-829b-82ed49dd67cd\") "
Sep 11 21:39:26.000752 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:26.000721    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da27985-0a5d-4a58-829b-82ed49dd67cd-kube-api-access-wj97g" (OuterVolumeSpecName: "kube-api-access-wj97g") pod "0da27985-0a5d-4a58-829b-82ed49dd67cd" (UID: "0da27985-0a5d-4a58-829b-82ed49dd67cd"). InnerVolumeSpecName "kube-api-access-wj97g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:39:26.099064 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:26.099032    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj97g\" (UniqueName: \"kubernetes.io/projected/0da27985-0a5d-4a58-829b-82ed49dd67cd-kube-api-access-wj97g\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:39:26.660371 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:26.660333    2544 generic.go:356] "Generic (PLEG): container finished" podID="0da27985-0a5d-4a58-829b-82ed49dd67cd" containerID="6a66ad9eedfe6c973734c61c70f36a97a8d62eca521d991ebfc8a4ddd420f88b" exitCode=0
Sep 11 21:39:26.660758 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:26.660375    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lbm7j"
Sep 11 21:39:26.660758 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:26.660420    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lbm7j" event={"ID":"0da27985-0a5d-4a58-829b-82ed49dd67cd","Type":"ContainerDied","Data":"6a66ad9eedfe6c973734c61c70f36a97a8d62eca521d991ebfc8a4ddd420f88b"}
Sep 11 21:39:26.660758 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:26.660457    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lbm7j" event={"ID":"0da27985-0a5d-4a58-829b-82ed49dd67cd","Type":"ContainerDied","Data":"ce51b869ac6786e0a89a84e82a719ca69f2c7362afbe347d5f97f62f695d34e7"}
Sep 11 21:39:26.660758 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:26.660475    2544 scope.go:117] "RemoveContainer" containerID="6a66ad9eedfe6c973734c61c70f36a97a8d62eca521d991ebfc8a4ddd420f88b"
Sep 11 21:39:26.668912 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:26.668899    2544 scope.go:117] "RemoveContainer" containerID="6a66ad9eedfe6c973734c61c70f36a97a8d62eca521d991ebfc8a4ddd420f88b"
Sep 11 21:39:26.669177 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:39:26.669155    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a66ad9eedfe6c973734c61c70f36a97a8d62eca521d991ebfc8a4ddd420f88b\": container with ID starting with 6a66ad9eedfe6c973734c61c70f36a97a8d62eca521d991ebfc8a4ddd420f88b not found: ID does not exist" containerID="6a66ad9eedfe6c973734c61c70f36a97a8d62eca521d991ebfc8a4ddd420f88b"
Sep 11 21:39:26.669264 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:26.669191    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a66ad9eedfe6c973734c61c70f36a97a8d62eca521d991ebfc8a4ddd420f88b"} err="failed to get container status \"6a66ad9eedfe6c973734c61c70f36a97a8d62eca521d991ebfc8a4ddd420f88b\": rpc error: code = NotFound desc = could not find container \"6a66ad9eedfe6c973734c61c70f36a97a8d62eca521d991ebfc8a4ddd420f88b\": container with ID starting with 6a66ad9eedfe6c973734c61c70f36a97a8d62eca521d991ebfc8a4ddd420f88b not found: ID does not exist"
Sep 11 21:39:26.680326 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:26.680301    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lbm7j"]
Sep 11 21:39:26.684007 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:26.683984    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lbm7j"]
Sep 11 21:39:27.591807 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:27.591774    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0da27985-0a5d-4a58-829b-82ed49dd67cd" path="/var/lib/kubelet/pods/0da27985-0a5d-4a58-829b-82ed49dd67cd/volumes"
Sep 11 21:39:47.380245 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.380198    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-586wm"]
Sep 11 21:39:47.380647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.380504    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da27985-0a5d-4a58-829b-82ed49dd67cd" containerName="authorino"
Sep 11 21:39:47.380647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.380517    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da27985-0a5d-4a58-829b-82ed49dd67cd" containerName="authorino"
Sep 11 21:39:47.380647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.380575    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da27985-0a5d-4a58-829b-82ed49dd67cd" containerName="authorino"
Sep 11 21:39:47.386708 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.386684    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-586wm"
Sep 11 21:39:47.392551 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.392524    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-586wm"]
Sep 11 21:39:47.439476 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.439439    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-586wm"]
Sep 11 21:39:47.439685 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:39:47.439667    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-jdqrj], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-586wm" podUID="df92c04d-14fd-480f-a699-8707af4ceaf9"
Sep 11 21:39:47.464372 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.464336    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdqrj\" (UniqueName: \"kubernetes.io/projected/df92c04d-14fd-480f-a699-8707af4ceaf9-kube-api-access-jdqrj\") pod \"authorino-557f876f69-586wm\" (UID: \"df92c04d-14fd-480f-a699-8707af4ceaf9\") " pod="kuadrant-system/authorino-557f876f69-586wm"
Sep 11 21:39:47.468847 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.468817    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6m8q4"]
Sep 11 21:39:47.472693 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.472674    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-6m8q4"
Sep 11 21:39:47.474890 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.474869    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 21:39:47.478531 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.478512    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6m8q4"]
Sep 11 21:39:47.565410 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.565368    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49z5v\" (UniqueName: \"kubernetes.io/projected/2d9ed695-9d34-401b-b868-faf59275f8a9-kube-api-access-49z5v\") pod \"authorino-6d4dfc649f-6m8q4\" (UID: \"2d9ed695-9d34-401b-b868-faf59275f8a9\") " pod="kuadrant-system/authorino-6d4dfc649f-6m8q4"
Sep 11 21:39:47.565581 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.565423    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdqrj\" (UniqueName: \"kubernetes.io/projected/df92c04d-14fd-480f-a699-8707af4ceaf9-kube-api-access-jdqrj\") pod \"authorino-557f876f69-586wm\" (UID: \"df92c04d-14fd-480f-a699-8707af4ceaf9\") " pod="kuadrant-system/authorino-557f876f69-586wm"
Sep 11 21:39:47.565581 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.565475    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d9ed695-9d34-401b-b868-faf59275f8a9-tls-cert\") pod \"authorino-6d4dfc649f-6m8q4\" (UID: \"2d9ed695-9d34-401b-b868-faf59275f8a9\") " pod="kuadrant-system/authorino-6d4dfc649f-6m8q4"
Sep 11 21:39:47.574818 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.574796    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdqrj\" (UniqueName: \"kubernetes.io/projected/df92c04d-14fd-480f-a699-8707af4ceaf9-kube-api-access-jdqrj\") pod \"authorino-557f876f69-586wm\" (UID: \"df92c04d-14fd-480f-a699-8707af4ceaf9\") " pod="kuadrant-system/authorino-557f876f69-586wm"
Sep 11 21:39:47.628485 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.628447    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6m8q4"]
Sep 11 21:39:47.628755 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:39:47.628732    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-49z5v tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-6m8q4" podUID="2d9ed695-9d34-401b-b868-faf59275f8a9"
Sep 11 21:39:47.661475 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.661401    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-85cbf77b49-vqtz8"]
Sep 11 21:39:47.665213 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.665196    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85cbf77b49-vqtz8"
Sep 11 21:39:47.665820 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.665792    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d9ed695-9d34-401b-b868-faf59275f8a9-tls-cert\") pod \"authorino-6d4dfc649f-6m8q4\" (UID: \"2d9ed695-9d34-401b-b868-faf59275f8a9\") " pod="kuadrant-system/authorino-6d4dfc649f-6m8q4"
Sep 11 21:39:47.665917 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.665872    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49z5v\" (UniqueName: \"kubernetes.io/projected/2d9ed695-9d34-401b-b868-faf59275f8a9-kube-api-access-49z5v\") pod \"authorino-6d4dfc649f-6m8q4\" (UID: \"2d9ed695-9d34-401b-b868-faf59275f8a9\") " pod="kuadrant-system/authorino-6d4dfc649f-6m8q4"
Sep 11 21:39:47.668250 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.668215    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d9ed695-9d34-401b-b868-faf59275f8a9-tls-cert\") pod \"authorino-6d4dfc649f-6m8q4\" (UID: \"2d9ed695-9d34-401b-b868-faf59275f8a9\") " pod="kuadrant-system/authorino-6d4dfc649f-6m8q4"
Sep 11 21:39:47.673040 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.673014    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85cbf77b49-vqtz8"]
Sep 11 21:39:47.675943 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.675923    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49z5v\" (UniqueName: \"kubernetes.io/projected/2d9ed695-9d34-401b-b868-faf59275f8a9-kube-api-access-49z5v\") pod \"authorino-6d4dfc649f-6m8q4\" (UID: \"2d9ed695-9d34-401b-b868-faf59275f8a9\") " pod="kuadrant-system/authorino-6d4dfc649f-6m8q4"
Sep 11 21:39:47.751472 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.751442    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-586wm"
Sep 11 21:39:47.751640 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.751454    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-6m8q4"
Sep 11 21:39:47.758595 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.758571    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-586wm"
Sep 11 21:39:47.761189 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.761174    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-6m8q4"
Sep 11 21:39:47.766697 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.766673    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvq5f\" (UniqueName: \"kubernetes.io/projected/65761e54-97e9-40bb-bb3f-e82ed43d3dc9-kube-api-access-cvq5f\") pod \"authorino-85cbf77b49-vqtz8\" (UID: \"65761e54-97e9-40bb-bb3f-e82ed43d3dc9\") " pod="kuadrant-system/authorino-85cbf77b49-vqtz8"
Sep 11 21:39:47.766809 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.766725    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65761e54-97e9-40bb-bb3f-e82ed43d3dc9-tls-cert\") pod \"authorino-85cbf77b49-vqtz8\" (UID: \"65761e54-97e9-40bb-bb3f-e82ed43d3dc9\") " pod="kuadrant-system/authorino-85cbf77b49-vqtz8"
Sep 11 21:39:47.867207 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.867174    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49z5v\" (UniqueName: \"kubernetes.io/projected/2d9ed695-9d34-401b-b868-faf59275f8a9-kube-api-access-49z5v\") pod \"2d9ed695-9d34-401b-b868-faf59275f8a9\" (UID: \"2d9ed695-9d34-401b-b868-faf59275f8a9\") "
Sep 11 21:39:47.867387 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.867263    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d9ed695-9d34-401b-b868-faf59275f8a9-tls-cert\") pod \"2d9ed695-9d34-401b-b868-faf59275f8a9\" (UID: \"2d9ed695-9d34-401b-b868-faf59275f8a9\") "
Sep 11 21:39:47.867387 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.867291    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdqrj\" (UniqueName: \"kubernetes.io/projected/df92c04d-14fd-480f-a699-8707af4ceaf9-kube-api-access-jdqrj\") pod \"df92c04d-14fd-480f-a699-8707af4ceaf9\" (UID: \"df92c04d-14fd-480f-a699-8707af4ceaf9\") "
Sep 11 21:39:47.867387 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.867376    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvq5f\" (UniqueName: \"kubernetes.io/projected/65761e54-97e9-40bb-bb3f-e82ed43d3dc9-kube-api-access-cvq5f\") pod \"authorino-85cbf77b49-vqtz8\" (UID: \"65761e54-97e9-40bb-bb3f-e82ed43d3dc9\") " pod="kuadrant-system/authorino-85cbf77b49-vqtz8"
Sep 11 21:39:47.867564 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.867426    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65761e54-97e9-40bb-bb3f-e82ed43d3dc9-tls-cert\") pod \"authorino-85cbf77b49-vqtz8\" (UID: \"65761e54-97e9-40bb-bb3f-e82ed43d3dc9\") " pod="kuadrant-system/authorino-85cbf77b49-vqtz8"
Sep 11 21:39:47.869707 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.869668    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d9ed695-9d34-401b-b868-faf59275f8a9-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "2d9ed695-9d34-401b-b868-faf59275f8a9" (UID: "2d9ed695-9d34-401b-b868-faf59275f8a9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:39:47.869707 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.869694    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9ed695-9d34-401b-b868-faf59275f8a9-kube-api-access-49z5v" (OuterVolumeSpecName: "kube-api-access-49z5v") pod "2d9ed695-9d34-401b-b868-faf59275f8a9" (UID: "2d9ed695-9d34-401b-b868-faf59275f8a9"). InnerVolumeSpecName "kube-api-access-49z5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:39:47.869908 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.869717    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df92c04d-14fd-480f-a699-8707af4ceaf9-kube-api-access-jdqrj" (OuterVolumeSpecName: "kube-api-access-jdqrj") pod "df92c04d-14fd-480f-a699-8707af4ceaf9" (UID: "df92c04d-14fd-480f-a699-8707af4ceaf9"). InnerVolumeSpecName "kube-api-access-jdqrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:39:47.869908 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.869849    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65761e54-97e9-40bb-bb3f-e82ed43d3dc9-tls-cert\") pod \"authorino-85cbf77b49-vqtz8\" (UID: \"65761e54-97e9-40bb-bb3f-e82ed43d3dc9\") " pod="kuadrant-system/authorino-85cbf77b49-vqtz8"
Sep 11 21:39:47.875765 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.875740    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvq5f\" (UniqueName: \"kubernetes.io/projected/65761e54-97e9-40bb-bb3f-e82ed43d3dc9-kube-api-access-cvq5f\") pod \"authorino-85cbf77b49-vqtz8\" (UID: \"65761e54-97e9-40bb-bb3f-e82ed43d3dc9\") " pod="kuadrant-system/authorino-85cbf77b49-vqtz8"
Sep 11 21:39:47.968133 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.968050    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49z5v\" (UniqueName: \"kubernetes.io/projected/2d9ed695-9d34-401b-b868-faf59275f8a9-kube-api-access-49z5v\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:39:47.968133 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.968082    2544 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d9ed695-9d34-401b-b868-faf59275f8a9-tls-cert\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:39:47.968133 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.968092    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdqrj\" (UniqueName: \"kubernetes.io/projected/df92c04d-14fd-480f-a699-8707af4ceaf9-kube-api-access-jdqrj\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:39:47.987969 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:47.987932    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85cbf77b49-vqtz8"
Sep 11 21:39:48.256590 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:48.256447    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85cbf77b49-vqtz8"]
Sep 11 21:39:48.259276 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:39:48.259250    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65761e54_97e9_40bb_bb3f_e82ed43d3dc9.slice/crio-e2132a8575a2ca963b909082cb3b2d867293ed4237b0cc980ec84c6995381ea6 WatchSource:0}: Error finding container e2132a8575a2ca963b909082cb3b2d867293ed4237b0cc980ec84c6995381ea6: Status 404 returned error can't find the container with id e2132a8575a2ca963b909082cb3b2d867293ed4237b0cc980ec84c6995381ea6
Sep 11 21:39:48.756995 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:48.756958    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85cbf77b49-vqtz8" event={"ID":"65761e54-97e9-40bb-bb3f-e82ed43d3dc9","Type":"ContainerStarted","Data":"e2132a8575a2ca963b909082cb3b2d867293ed4237b0cc980ec84c6995381ea6"}
Sep 11 21:39:48.756995 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:48.756990    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-586wm"
Sep 11 21:39:48.757415 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:48.757166    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-6m8q4"
Sep 11 21:39:48.787285 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:48.787257    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-586wm"]
Sep 11 21:39:48.788952 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:48.788929    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-586wm"]
Sep 11 21:39:48.811887 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:48.811860    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6m8q4"]
Sep 11 21:39:48.815623 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:48.815598    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6m8q4"]
Sep 11 21:39:49.595171 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:49.595141    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d9ed695-9d34-401b-b868-faf59275f8a9" path="/var/lib/kubelet/pods/2d9ed695-9d34-401b-b868-faf59275f8a9/volumes"
Sep 11 21:39:49.595377 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:49.595363    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df92c04d-14fd-480f-a699-8707af4ceaf9" path="/var/lib/kubelet/pods/df92c04d-14fd-480f-a699-8707af4ceaf9/volumes"
Sep 11 21:39:49.762214 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:49.762173    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85cbf77b49-vqtz8" event={"ID":"65761e54-97e9-40bb-bb3f-e82ed43d3dc9","Type":"ContainerStarted","Data":"d58e006293c43c90469bed11522af8e82ad94d4849974bf82926658bd63d75c2"}
Sep 11 21:39:49.778397 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:49.778345    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-85cbf77b49-vqtz8" podStartSLOduration=2.270053673 podStartE2EDuration="2.77832818s" totalImagesPullingTime="508.274507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:39:49.777004708 +0000 UTC m=+552.746421528" watchObservedRunningTime="2026-09-11 21:39:49.77832818 +0000 UTC m=+552.747744999"
Sep 11 21:39:49.801495 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:49.801457    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-j9kgc"]
Sep 11 21:39:49.802699 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:49.801675    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-j9kgc" podUID="26400880-a218-4304-8e8d-7b9875c66310" containerName="authorino" containerID="cri-o://286763dee525cf3898c509886418ca34d1858d31fcb6f336a69584575524af3e" gracePeriod=30
Sep 11 21:39:50.053085 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:50.053060    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-j9kgc"
Sep 11 21:39:50.186898 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:50.186813    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zckfv\" (UniqueName: \"kubernetes.io/projected/26400880-a218-4304-8e8d-7b9875c66310-kube-api-access-zckfv\") pod \"26400880-a218-4304-8e8d-7b9875c66310\" (UID: \"26400880-a218-4304-8e8d-7b9875c66310\") "
Sep 11 21:39:50.188827 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:50.188798    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26400880-a218-4304-8e8d-7b9875c66310-kube-api-access-zckfv" (OuterVolumeSpecName: "kube-api-access-zckfv") pod "26400880-a218-4304-8e8d-7b9875c66310" (UID: "26400880-a218-4304-8e8d-7b9875c66310"). InnerVolumeSpecName "kube-api-access-zckfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:39:50.288126 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:50.288086    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zckfv\" (UniqueName: \"kubernetes.io/projected/26400880-a218-4304-8e8d-7b9875c66310-kube-api-access-zckfv\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:39:50.767731 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:50.767697    2544 generic.go:356] "Generic (PLEG): container finished" podID="26400880-a218-4304-8e8d-7b9875c66310" containerID="286763dee525cf3898c509886418ca34d1858d31fcb6f336a69584575524af3e" exitCode=0
Sep 11 21:39:50.768163 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:50.767739    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-j9kgc"
Sep 11 21:39:50.768163 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:50.767766    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-j9kgc" event={"ID":"26400880-a218-4304-8e8d-7b9875c66310","Type":"ContainerDied","Data":"286763dee525cf3898c509886418ca34d1858d31fcb6f336a69584575524af3e"}
Sep 11 21:39:50.768163 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:50.767811    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-j9kgc" event={"ID":"26400880-a218-4304-8e8d-7b9875c66310","Type":"ContainerDied","Data":"0671fab8f3b7a7323a116a3116eb3d6868e8a64f38170e364bf9002e9078ea16"}
Sep 11 21:39:50.768163 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:50.767828    2544 scope.go:117] "RemoveContainer" containerID="286763dee525cf3898c509886418ca34d1858d31fcb6f336a69584575524af3e"
Sep 11 21:39:50.775872 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:50.775853    2544 scope.go:117] "RemoveContainer" containerID="286763dee525cf3898c509886418ca34d1858d31fcb6f336a69584575524af3e"
Sep 11 21:39:50.776144 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:39:50.776126    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"286763dee525cf3898c509886418ca34d1858d31fcb6f336a69584575524af3e\": container with ID starting with 286763dee525cf3898c509886418ca34d1858d31fcb6f336a69584575524af3e not found: ID does not exist" containerID="286763dee525cf3898c509886418ca34d1858d31fcb6f336a69584575524af3e"
Sep 11 21:39:50.776191 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:50.776154    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286763dee525cf3898c509886418ca34d1858d31fcb6f336a69584575524af3e"} err="failed to get container status \"286763dee525cf3898c509886418ca34d1858d31fcb6f336a69584575524af3e\": rpc error: code = NotFound desc = could not find container \"286763dee525cf3898c509886418ca34d1858d31fcb6f336a69584575524af3e\": container with ID starting with 286763dee525cf3898c509886418ca34d1858d31fcb6f336a69584575524af3e not found: ID does not exist"
Sep 11 21:39:50.787842 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:50.787815    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-j9kgc"]
Sep 11 21:39:50.792286 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:50.792260    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-j9kgc"]
Sep 11 21:39:51.590929 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:39:51.590898    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26400880-a218-4304-8e8d-7b9875c66310" path="/var/lib/kubelet/pods/26400880-a218-4304-8e8d-7b9875c66310/volumes"
Sep 11 21:40:54.869360 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.869326    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"]
Sep 11 21:40:54.869846 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.869616    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26400880-a218-4304-8e8d-7b9875c66310" containerName="authorino"
Sep 11 21:40:54.869846 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.869627    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="26400880-a218-4304-8e8d-7b9875c66310" containerName="authorino"
Sep 11 21:40:54.869846 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.869686    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="26400880-a218-4304-8e8d-7b9875c66310" containerName="authorino"
Sep 11 21:40:54.873656 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.873636    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:54.877096 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.877073    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 21:40:54.877359 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.877076    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 11 21:40:54.877472 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.877087    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 21:40:54.877472 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.877108    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-442fp\""
Sep 11 21:40:54.882195 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.882173    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"]
Sep 11 21:40:54.887070 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.887048    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8c695c0-738f-4132-8cfb-25e62b7e8c2d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7\" (UID: \"c8c695c0-738f-4132-8cfb-25e62b7e8c2d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:54.887167 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.887096    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8c695c0-738f-4132-8cfb-25e62b7e8c2d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7\" (UID: \"c8c695c0-738f-4132-8cfb-25e62b7e8c2d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:54.887167 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.887121    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2925\" (UniqueName: \"kubernetes.io/projected/c8c695c0-738f-4132-8cfb-25e62b7e8c2d-kube-api-access-x2925\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7\" (UID: \"c8c695c0-738f-4132-8cfb-25e62b7e8c2d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:54.887167 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.887161    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8c695c0-738f-4132-8cfb-25e62b7e8c2d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7\" (UID: \"c8c695c0-738f-4132-8cfb-25e62b7e8c2d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:54.887335 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.887295    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8c695c0-738f-4132-8cfb-25e62b7e8c2d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7\" (UID: \"c8c695c0-738f-4132-8cfb-25e62b7e8c2d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:54.887335 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.887326    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8c695c0-738f-4132-8cfb-25e62b7e8c2d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7\" (UID: \"c8c695c0-738f-4132-8cfb-25e62b7e8c2d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:54.987737 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.987702    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8c695c0-738f-4132-8cfb-25e62b7e8c2d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7\" (UID: \"c8c695c0-738f-4132-8cfb-25e62b7e8c2d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:54.987912 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.987745    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8c695c0-738f-4132-8cfb-25e62b7e8c2d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7\" (UID: \"c8c695c0-738f-4132-8cfb-25e62b7e8c2d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:54.987912 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.987763    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2925\" (UniqueName: \"kubernetes.io/projected/c8c695c0-738f-4132-8cfb-25e62b7e8c2d-kube-api-access-x2925\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7\" (UID: \"c8c695c0-738f-4132-8cfb-25e62b7e8c2d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:54.987912 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.987785    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8c695c0-738f-4132-8cfb-25e62b7e8c2d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7\" (UID: \"c8c695c0-738f-4132-8cfb-25e62b7e8c2d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:54.987912 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.987833    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8c695c0-738f-4132-8cfb-25e62b7e8c2d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7\" (UID: \"c8c695c0-738f-4132-8cfb-25e62b7e8c2d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:54.987912 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.987852    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8c695c0-738f-4132-8cfb-25e62b7e8c2d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7\" (UID: \"c8c695c0-738f-4132-8cfb-25e62b7e8c2d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:54.988208 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.988185    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8c695c0-738f-4132-8cfb-25e62b7e8c2d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7\" (UID: \"c8c695c0-738f-4132-8cfb-25e62b7e8c2d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:54.988208 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.988197    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8c695c0-738f-4132-8cfb-25e62b7e8c2d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7\" (UID: \"c8c695c0-738f-4132-8cfb-25e62b7e8c2d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:54.988315 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.988273    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8c695c0-738f-4132-8cfb-25e62b7e8c2d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7\" (UID: \"c8c695c0-738f-4132-8cfb-25e62b7e8c2d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:54.990116 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.990093    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8c695c0-738f-4132-8cfb-25e62b7e8c2d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7\" (UID: \"c8c695c0-738f-4132-8cfb-25e62b7e8c2d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:54.990293 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.990275    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8c695c0-738f-4132-8cfb-25e62b7e8c2d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7\" (UID: \"c8c695c0-738f-4132-8cfb-25e62b7e8c2d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:54.995346 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:54.995328    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2925\" (UniqueName: \"kubernetes.io/projected/c8c695c0-738f-4132-8cfb-25e62b7e8c2d-kube-api-access-x2925\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7\" (UID: \"c8c695c0-738f-4132-8cfb-25e62b7e8c2d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:55.184586 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:55.184501    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:40:55.461858 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:55.461769    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"]
Sep 11 21:40:55.464743 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:40:55.464715    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c695c0_738f_4132_8cfb_25e62b7e8c2d.slice/crio-029351b9d4ca49240b2583f75dd8e46dd43857aa3d35f6c5a1cee631ca3f460a WatchSource:0}: Error finding container 029351b9d4ca49240b2583f75dd8e46dd43857aa3d35f6c5a1cee631ca3f460a: Status 404 returned error can't find the container with id 029351b9d4ca49240b2583f75dd8e46dd43857aa3d35f6c5a1cee631ca3f460a
Sep 11 21:40:56.036151 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:56.036103    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7" event={"ID":"c8c695c0-738f-4132-8cfb-25e62b7e8c2d","Type":"ContainerStarted","Data":"029351b9d4ca49240b2583f75dd8e46dd43857aa3d35f6c5a1cee631ca3f460a"}
Sep 11 21:40:57.041572 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:57.041538    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7" event={"ID":"c8c695c0-738f-4132-8cfb-25e62b7e8c2d","Type":"ContainerStarted","Data":"8a8e919c588adeb9665dccf9ec9ea8685e1680468714ab157550944eb56ed929"}
Sep 11 21:40:57.059104 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:40:57.059050    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7" podStartSLOduration=1.963875955 podStartE2EDuration="3.059032505s" totalImagesPullingTime="1.09515655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:40:57.05718061 +0000 UTC m=+620.026597428" watchObservedRunningTime="2026-09-11 21:40:57.059032505 +0000 UTC m=+620.028449323"
Sep 11 21:41:05.184918 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:05.184872    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:41:05.187502 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:05.184938    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:41:05.197267 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:05.197244    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:41:06.089461 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:06.089426    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7"
Sep 11 21:41:23.865157 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:23.865120    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"]
Sep 11 21:41:23.868288 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:23.868271    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:23.870739 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:23.870720    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 11 21:41:23.879425 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:23.879391    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"]
Sep 11 21:41:23.933965 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:23.933928    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f0af874-2871-4f99-b7f9-0506f9bcf904-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp\" (UID: \"7f0af874-2871-4f99-b7f9-0506f9bcf904\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:23.934153 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:23.933986    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd7gk\" (UniqueName: \"kubernetes.io/projected/7f0af874-2871-4f99-b7f9-0506f9bcf904-kube-api-access-bd7gk\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp\" (UID: \"7f0af874-2871-4f99-b7f9-0506f9bcf904\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:23.934153 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:23.934030    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f0af874-2871-4f99-b7f9-0506f9bcf904-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp\" (UID: \"7f0af874-2871-4f99-b7f9-0506f9bcf904\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:23.934153 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:23.934120    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f0af874-2871-4f99-b7f9-0506f9bcf904-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp\" (UID: \"7f0af874-2871-4f99-b7f9-0506f9bcf904\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:23.934317 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:23.934172    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f0af874-2871-4f99-b7f9-0506f9bcf904-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp\" (UID: \"7f0af874-2871-4f99-b7f9-0506f9bcf904\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:23.934317 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:23.934194    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f0af874-2871-4f99-b7f9-0506f9bcf904-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp\" (UID: \"7f0af874-2871-4f99-b7f9-0506f9bcf904\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:24.035515 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:24.035472    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f0af874-2871-4f99-b7f9-0506f9bcf904-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp\" (UID: \"7f0af874-2871-4f99-b7f9-0506f9bcf904\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:24.035681 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:24.035520    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f0af874-2871-4f99-b7f9-0506f9bcf904-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp\" (UID: \"7f0af874-2871-4f99-b7f9-0506f9bcf904\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:24.035681 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:24.035549    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f0af874-2871-4f99-b7f9-0506f9bcf904-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp\" (UID: \"7f0af874-2871-4f99-b7f9-0506f9bcf904\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:24.035681 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:24.035596    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f0af874-2871-4f99-b7f9-0506f9bcf904-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp\" (UID: \"7f0af874-2871-4f99-b7f9-0506f9bcf904\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:24.035681 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:24.035627    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd7gk\" (UniqueName: \"kubernetes.io/projected/7f0af874-2871-4f99-b7f9-0506f9bcf904-kube-api-access-bd7gk\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp\" (UID: \"7f0af874-2871-4f99-b7f9-0506f9bcf904\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:24.035681 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:24.035662    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f0af874-2871-4f99-b7f9-0506f9bcf904-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp\" (UID: \"7f0af874-2871-4f99-b7f9-0506f9bcf904\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:24.036008 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:24.035980    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f0af874-2871-4f99-b7f9-0506f9bcf904-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp\" (UID: \"7f0af874-2871-4f99-b7f9-0506f9bcf904\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:24.036083 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:24.036018    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f0af874-2871-4f99-b7f9-0506f9bcf904-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp\" (UID: \"7f0af874-2871-4f99-b7f9-0506f9bcf904\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:24.036083 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:24.036073    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f0af874-2871-4f99-b7f9-0506f9bcf904-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp\" (UID: \"7f0af874-2871-4f99-b7f9-0506f9bcf904\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:24.037664 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:24.037645    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f0af874-2871-4f99-b7f9-0506f9bcf904-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp\" (UID: \"7f0af874-2871-4f99-b7f9-0506f9bcf904\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:24.038050 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:24.038030    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f0af874-2871-4f99-b7f9-0506f9bcf904-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp\" (UID: \"7f0af874-2871-4f99-b7f9-0506f9bcf904\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:24.047414 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:24.047386    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd7gk\" (UniqueName: \"kubernetes.io/projected/7f0af874-2871-4f99-b7f9-0506f9bcf904-kube-api-access-bd7gk\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp\" (UID: \"7f0af874-2871-4f99-b7f9-0506f9bcf904\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:24.178499 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:24.178414    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:24.448251 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:24.448211    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"]
Sep 11 21:41:24.450338 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:41:24.450308    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0af874_2871_4f99_b7f9_0506f9bcf904.slice/crio-df360d9d2cbc9a285ec330e80a9691a18a83ca3d9eba788c1bbe1059de6fd7f3 WatchSource:0}: Error finding container df360d9d2cbc9a285ec330e80a9691a18a83ca3d9eba788c1bbe1059de6fd7f3: Status 404 returned error can't find the container with id df360d9d2cbc9a285ec330e80a9691a18a83ca3d9eba788c1bbe1059de6fd7f3
Sep 11 21:41:24.451638 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:24.451623    2544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:41:25.156363 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:25.156325    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp" event={"ID":"7f0af874-2871-4f99-b7f9-0506f9bcf904","Type":"ContainerStarted","Data":"827a5372b909465a9de0982f0de8d25d2146b26963b7ca2f45e5b13891ee5a1f"}
Sep 11 21:41:25.156784 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:25.156380    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp" event={"ID":"7f0af874-2871-4f99-b7f9-0506f9bcf904","Type":"ContainerStarted","Data":"df360d9d2cbc9a285ec330e80a9691a18a83ca3d9eba788c1bbe1059de6fd7f3"}
Sep 11 21:41:25.172545 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:25.172481    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp" podStartSLOduration=1.988893711 podStartE2EDuration="2.172459484s" totalImagesPullingTime="183.565773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:41:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:41:25.172006413 +0000 UTC m=+648.141423233" watchObservedRunningTime="2026-09-11 21:41:25.172459484 +0000 UTC m=+648.141876303"
Sep 11 21:41:30.962967 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:30.962933    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"]
Sep 11 21:41:30.967296 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:30.967277    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:30.969562 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:30.969543    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 11 21:41:30.974534 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:30.974514    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"]
Sep 11 21:41:31.093292 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.093245    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62efe647-ce65-46b7-8937-3b7f534fa52b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7nqcx\" (UID: \"62efe647-ce65-46b7-8937-3b7f534fa52b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.093473 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.093306    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62efe647-ce65-46b7-8937-3b7f534fa52b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7nqcx\" (UID: \"62efe647-ce65-46b7-8937-3b7f534fa52b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.093473 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.093366    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62efe647-ce65-46b7-8937-3b7f534fa52b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7nqcx\" (UID: \"62efe647-ce65-46b7-8937-3b7f534fa52b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.093473 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.093406    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkzw4\" (UniqueName: \"kubernetes.io/projected/62efe647-ce65-46b7-8937-3b7f534fa52b-kube-api-access-jkzw4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7nqcx\" (UID: \"62efe647-ce65-46b7-8937-3b7f534fa52b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.093589 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.093473    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62efe647-ce65-46b7-8937-3b7f534fa52b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7nqcx\" (UID: \"62efe647-ce65-46b7-8937-3b7f534fa52b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.093589 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.093493    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62efe647-ce65-46b7-8937-3b7f534fa52b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7nqcx\" (UID: \"62efe647-ce65-46b7-8937-3b7f534fa52b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.194458 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.194420    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62efe647-ce65-46b7-8937-3b7f534fa52b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7nqcx\" (UID: \"62efe647-ce65-46b7-8937-3b7f534fa52b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.194458 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.194460    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62efe647-ce65-46b7-8937-3b7f534fa52b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7nqcx\" (UID: \"62efe647-ce65-46b7-8937-3b7f534fa52b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.194673 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.194514    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62efe647-ce65-46b7-8937-3b7f534fa52b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7nqcx\" (UID: \"62efe647-ce65-46b7-8937-3b7f534fa52b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.194673 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.194546    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62efe647-ce65-46b7-8937-3b7f534fa52b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7nqcx\" (UID: \"62efe647-ce65-46b7-8937-3b7f534fa52b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.194673 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.194576    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62efe647-ce65-46b7-8937-3b7f534fa52b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7nqcx\" (UID: \"62efe647-ce65-46b7-8937-3b7f534fa52b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.194673 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.194605    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkzw4\" (UniqueName: \"kubernetes.io/projected/62efe647-ce65-46b7-8937-3b7f534fa52b-kube-api-access-jkzw4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7nqcx\" (UID: \"62efe647-ce65-46b7-8937-3b7f534fa52b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.194985 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.194961    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62efe647-ce65-46b7-8937-3b7f534fa52b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7nqcx\" (UID: \"62efe647-ce65-46b7-8937-3b7f534fa52b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.195027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.194973    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62efe647-ce65-46b7-8937-3b7f534fa52b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7nqcx\" (UID: \"62efe647-ce65-46b7-8937-3b7f534fa52b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.195074 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.195055    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62efe647-ce65-46b7-8937-3b7f534fa52b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7nqcx\" (UID: \"62efe647-ce65-46b7-8937-3b7f534fa52b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.196832 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.196815    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62efe647-ce65-46b7-8937-3b7f534fa52b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7nqcx\" (UID: \"62efe647-ce65-46b7-8937-3b7f534fa52b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.197098 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.197080    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62efe647-ce65-46b7-8937-3b7f534fa52b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7nqcx\" (UID: \"62efe647-ce65-46b7-8937-3b7f534fa52b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.202195 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.202173    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkzw4\" (UniqueName: \"kubernetes.io/projected/62efe647-ce65-46b7-8937-3b7f534fa52b-kube-api-access-jkzw4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7nqcx\" (UID: \"62efe647-ce65-46b7-8937-3b7f534fa52b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.278398 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.278358    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:31.550789 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:31.550763    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"]
Sep 11 21:41:31.552971 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:41:31.552940    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62efe647_ce65_46b7_8937_3b7f534fa52b.slice/crio-5b27602fc661890c05be2a96bbc2b9882ad6cef3e2f9dcb9818546189c69ff9e WatchSource:0}: Error finding container 5b27602fc661890c05be2a96bbc2b9882ad6cef3e2f9dcb9818546189c69ff9e: Status 404 returned error can't find the container with id 5b27602fc661890c05be2a96bbc2b9882ad6cef3e2f9dcb9818546189c69ff9e
Sep 11 21:41:32.188127 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:32.188081    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx" event={"ID":"62efe647-ce65-46b7-8937-3b7f534fa52b","Type":"ContainerStarted","Data":"9cc40a1bc827687264e15b9754909c3f5cf9d3370ed4494e42686a3da11ca654"}
Sep 11 21:41:32.188542 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:32.188151    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx" event={"ID":"62efe647-ce65-46b7-8937-3b7f534fa52b","Type":"ContainerStarted","Data":"5b27602fc661890c05be2a96bbc2b9882ad6cef3e2f9dcb9818546189c69ff9e"}
Sep 11 21:41:32.209100 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:32.209021    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx" podStartSLOduration=2.02281635 podStartE2EDuration="2.208998343s" totalImagesPullingTime="186.181993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:41:32.207404666 +0000 UTC m=+655.176821476" watchObservedRunningTime="2026-09-11 21:41:32.208998343 +0000 UTC m=+655.178415163"
Sep 11 21:41:34.179467 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:34.179425    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:34.179914 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:34.179485    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:34.191648 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:34.191616    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:34.205849 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:34.205824    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp"
Sep 11 21:41:41.279342 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:41.279298    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:41.279722 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:41.279367    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:41.291595 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:41.291561    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:41:42.237509 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:41:42.237482    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7nqcx"
Sep 11 21:42:04.252007 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.251974    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"]
Sep 11 21:42:04.255086 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.255068    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.257668 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.257643    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 21:42:04.266038 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.266011    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"]
Sep 11 21:42:04.361546 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.361504    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46b1d00-c254-4a5a-8adb-857e1aee0b25-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xltkj\" (UID: \"a46b1d00-c254-4a5a-8adb-857e1aee0b25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.361724 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.361555    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46b1d00-c254-4a5a-8adb-857e1aee0b25-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xltkj\" (UID: \"a46b1d00-c254-4a5a-8adb-857e1aee0b25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.361724 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.361646    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46b1d00-c254-4a5a-8adb-857e1aee0b25-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xltkj\" (UID: \"a46b1d00-c254-4a5a-8adb-857e1aee0b25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.361724 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.361674    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46b1d00-c254-4a5a-8adb-857e1aee0b25-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xltkj\" (UID: \"a46b1d00-c254-4a5a-8adb-857e1aee0b25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.361724 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.361695    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzll2\" (UniqueName: \"kubernetes.io/projected/a46b1d00-c254-4a5a-8adb-857e1aee0b25-kube-api-access-tzll2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xltkj\" (UID: \"a46b1d00-c254-4a5a-8adb-857e1aee0b25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.361952 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.361735    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46b1d00-c254-4a5a-8adb-857e1aee0b25-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xltkj\" (UID: \"a46b1d00-c254-4a5a-8adb-857e1aee0b25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.462954 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.462920    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46b1d00-c254-4a5a-8adb-857e1aee0b25-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xltkj\" (UID: \"a46b1d00-c254-4a5a-8adb-857e1aee0b25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.463118 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.462961    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46b1d00-c254-4a5a-8adb-857e1aee0b25-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xltkj\" (UID: \"a46b1d00-c254-4a5a-8adb-857e1aee0b25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.463118 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.463004    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46b1d00-c254-4a5a-8adb-857e1aee0b25-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xltkj\" (UID: \"a46b1d00-c254-4a5a-8adb-857e1aee0b25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.463118 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.463018    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46b1d00-c254-4a5a-8adb-857e1aee0b25-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xltkj\" (UID: \"a46b1d00-c254-4a5a-8adb-857e1aee0b25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.463118 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.463035    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzll2\" (UniqueName: \"kubernetes.io/projected/a46b1d00-c254-4a5a-8adb-857e1aee0b25-kube-api-access-tzll2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xltkj\" (UID: \"a46b1d00-c254-4a5a-8adb-857e1aee0b25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.463118 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.463050    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46b1d00-c254-4a5a-8adb-857e1aee0b25-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xltkj\" (UID: \"a46b1d00-c254-4a5a-8adb-857e1aee0b25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.463519 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.463478    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46b1d00-c254-4a5a-8adb-857e1aee0b25-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xltkj\" (UID: \"a46b1d00-c254-4a5a-8adb-857e1aee0b25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.463561 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.463528    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46b1d00-c254-4a5a-8adb-857e1aee0b25-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xltkj\" (UID: \"a46b1d00-c254-4a5a-8adb-857e1aee0b25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.463595 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.463578    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46b1d00-c254-4a5a-8adb-857e1aee0b25-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xltkj\" (UID: \"a46b1d00-c254-4a5a-8adb-857e1aee0b25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.465404 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.465378    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46b1d00-c254-4a5a-8adb-857e1aee0b25-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xltkj\" (UID: \"a46b1d00-c254-4a5a-8adb-857e1aee0b25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.465625 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.465610    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46b1d00-c254-4a5a-8adb-857e1aee0b25-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xltkj\" (UID: \"a46b1d00-c254-4a5a-8adb-857e1aee0b25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.471139 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.471118    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzll2\" (UniqueName: \"kubernetes.io/projected/a46b1d00-c254-4a5a-8adb-857e1aee0b25-kube-api-access-tzll2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xltkj\" (UID: \"a46b1d00-c254-4a5a-8adb-857e1aee0b25\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.563508 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.563406    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:04.862989 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:04.862909    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"]
Sep 11 21:42:04.865995 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:42:04.865961    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda46b1d00_c254_4a5a_8adb_857e1aee0b25.slice/crio-ae1a7d17b62efc3d9f6c860c14583fe4e471a266a077e8c210d08bdd70e20442 WatchSource:0}: Error finding container ae1a7d17b62efc3d9f6c860c14583fe4e471a266a077e8c210d08bdd70e20442: Status 404 returned error can't find the container with id ae1a7d17b62efc3d9f6c860c14583fe4e471a266a077e8c210d08bdd70e20442
Sep 11 21:42:05.323144 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:05.323102    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj" event={"ID":"a46b1d00-c254-4a5a-8adb-857e1aee0b25","Type":"ContainerStarted","Data":"22afe30d2e4f371dbe94853499c74f32b93e1bcdd25546105ca4e5d52a1fc588"}
Sep 11 21:42:05.323144 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:05.323151    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj" event={"ID":"a46b1d00-c254-4a5a-8adb-857e1aee0b25","Type":"ContainerStarted","Data":"ae1a7d17b62efc3d9f6c860c14583fe4e471a266a077e8c210d08bdd70e20442"}
Sep 11 21:42:05.341199 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:05.341152    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj" podStartSLOduration=1.171610486 podStartE2EDuration="1.341136432s" totalImagesPullingTime="169.525946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:42:05.339201677 +0000 UTC m=+688.308618494" watchObservedRunningTime="2026-09-11 21:42:05.341136432 +0000 UTC m=+688.310553293"
Sep 11 21:42:14.563897 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:14.563859    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:14.564452 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:14.563912    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:14.576357 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:14.576332    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:42:15.375660 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:42:15.375631    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xltkj"
Sep 11 21:43:20.940919 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:20.940873    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-85c7cf647b-mj2pd"]
Sep 11 21:43:20.944211 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:20.944186    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85c7cf647b-mj2pd"
Sep 11 21:43:20.951761 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:20.951737    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85c7cf647b-mj2pd"]
Sep 11 21:43:21.021097 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:21.021052    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67x7j\" (UniqueName: \"kubernetes.io/projected/457c6b3b-046a-401b-9027-2a4355a5c253-kube-api-access-67x7j\") pod \"authorino-85c7cf647b-mj2pd\" (UID: \"457c6b3b-046a-401b-9027-2a4355a5c253\") " pod="kuadrant-system/authorino-85c7cf647b-mj2pd"
Sep 11 21:43:21.021311 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:21.021105    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/457c6b3b-046a-401b-9027-2a4355a5c253-tls-cert\") pod \"authorino-85c7cf647b-mj2pd\" (UID: \"457c6b3b-046a-401b-9027-2a4355a5c253\") " pod="kuadrant-system/authorino-85c7cf647b-mj2pd"
Sep 11 21:43:21.122520 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:21.122487    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67x7j\" (UniqueName: \"kubernetes.io/projected/457c6b3b-046a-401b-9027-2a4355a5c253-kube-api-access-67x7j\") pod \"authorino-85c7cf647b-mj2pd\" (UID: \"457c6b3b-046a-401b-9027-2a4355a5c253\") " pod="kuadrant-system/authorino-85c7cf647b-mj2pd"
Sep 11 21:43:21.122690 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:21.122532    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/457c6b3b-046a-401b-9027-2a4355a5c253-tls-cert\") pod \"authorino-85c7cf647b-mj2pd\" (UID: \"457c6b3b-046a-401b-9027-2a4355a5c253\") " pod="kuadrant-system/authorino-85c7cf647b-mj2pd"
Sep 11 21:43:21.124951 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:21.124929    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/457c6b3b-046a-401b-9027-2a4355a5c253-tls-cert\") pod \"authorino-85c7cf647b-mj2pd\" (UID: \"457c6b3b-046a-401b-9027-2a4355a5c253\") " pod="kuadrant-system/authorino-85c7cf647b-mj2pd"
Sep 11 21:43:21.129843 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:21.129818    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67x7j\" (UniqueName: \"kubernetes.io/projected/457c6b3b-046a-401b-9027-2a4355a5c253-kube-api-access-67x7j\") pod \"authorino-85c7cf647b-mj2pd\" (UID: \"457c6b3b-046a-401b-9027-2a4355a5c253\") " pod="kuadrant-system/authorino-85c7cf647b-mj2pd"
Sep 11 21:43:21.253783 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:21.253752    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85c7cf647b-mj2pd"
Sep 11 21:43:21.522312 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:21.522282    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85c7cf647b-mj2pd"]
Sep 11 21:43:21.525501 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:43:21.525473    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod457c6b3b_046a_401b_9027_2a4355a5c253.slice/crio-8e859fa69d6d43eb57b5a7b0a180cf5811bf299377e40ab6846c589d6649424d WatchSource:0}: Error finding container 8e859fa69d6d43eb57b5a7b0a180cf5811bf299377e40ab6846c589d6649424d: Status 404 returned error can't find the container with id 8e859fa69d6d43eb57b5a7b0a180cf5811bf299377e40ab6846c589d6649424d
Sep 11 21:43:21.639629 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:21.639595    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85c7cf647b-mj2pd" event={"ID":"457c6b3b-046a-401b-9027-2a4355a5c253","Type":"ContainerStarted","Data":"8e859fa69d6d43eb57b5a7b0a180cf5811bf299377e40ab6846c589d6649424d"}
Sep 11 21:43:22.646922 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:22.646839    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85c7cf647b-mj2pd" event={"ID":"457c6b3b-046a-401b-9027-2a4355a5c253","Type":"ContainerStarted","Data":"0d5c05d6a922dce843227f7db0b621d3cbf5fcdcb427230651af07cd3eefb7d5"}
Sep 11 21:43:22.703243 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:22.701665    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-85c7cf647b-mj2pd" podStartSLOduration=1.9765201399999999 podStartE2EDuration="2.701639938s" totalImagesPullingTime="725.119798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:43:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:22.663637658 +0000 UTC m=+765.633054481" watchObservedRunningTime="2026-09-11 21:43:22.701639938 +0000 UTC m=+765.671056763"
Sep 11 21:43:22.705467 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:22.704073    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85cbf77b49-vqtz8"]
Sep 11 21:43:22.707183 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:22.707123    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-85cbf77b49-vqtz8" podUID="65761e54-97e9-40bb-bb3f-e82ed43d3dc9" containerName="authorino" containerID="cri-o://d58e006293c43c90469bed11522af8e82ad94d4849974bf82926658bd63d75c2" gracePeriod=30
Sep 11 21:43:22.959761 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:22.959735    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85cbf77b49-vqtz8"
Sep 11 21:43:23.040422 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:23.040382    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65761e54-97e9-40bb-bb3f-e82ed43d3dc9-tls-cert\") pod \"65761e54-97e9-40bb-bb3f-e82ed43d3dc9\" (UID: \"65761e54-97e9-40bb-bb3f-e82ed43d3dc9\") "
Sep 11 21:43:23.040607 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:23.040479    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvq5f\" (UniqueName: \"kubernetes.io/projected/65761e54-97e9-40bb-bb3f-e82ed43d3dc9-kube-api-access-cvq5f\") pod \"65761e54-97e9-40bb-bb3f-e82ed43d3dc9\" (UID: \"65761e54-97e9-40bb-bb3f-e82ed43d3dc9\") "
Sep 11 21:43:23.042655 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:23.042621    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65761e54-97e9-40bb-bb3f-e82ed43d3dc9-kube-api-access-cvq5f" (OuterVolumeSpecName: "kube-api-access-cvq5f") pod "65761e54-97e9-40bb-bb3f-e82ed43d3dc9" (UID: "65761e54-97e9-40bb-bb3f-e82ed43d3dc9"). InnerVolumeSpecName "kube-api-access-cvq5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:43:23.050621 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:23.050591    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65761e54-97e9-40bb-bb3f-e82ed43d3dc9-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "65761e54-97e9-40bb-bb3f-e82ed43d3dc9" (UID: "65761e54-97e9-40bb-bb3f-e82ed43d3dc9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:43:23.141158 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:23.141124    2544 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65761e54-97e9-40bb-bb3f-e82ed43d3dc9-tls-cert\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:43:23.141158 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:23.141155    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvq5f\" (UniqueName: \"kubernetes.io/projected/65761e54-97e9-40bb-bb3f-e82ed43d3dc9-kube-api-access-cvq5f\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:43:23.653035 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:23.653000    2544 generic.go:356] "Generic (PLEG): container finished" podID="65761e54-97e9-40bb-bb3f-e82ed43d3dc9" containerID="d58e006293c43c90469bed11522af8e82ad94d4849974bf82926658bd63d75c2" exitCode=0
Sep 11 21:43:23.653524 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:23.653046    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85cbf77b49-vqtz8"
Sep 11 21:43:23.653524 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:23.653091    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85cbf77b49-vqtz8" event={"ID":"65761e54-97e9-40bb-bb3f-e82ed43d3dc9","Type":"ContainerDied","Data":"d58e006293c43c90469bed11522af8e82ad94d4849974bf82926658bd63d75c2"}
Sep 11 21:43:23.653524 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:23.653138    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85cbf77b49-vqtz8" event={"ID":"65761e54-97e9-40bb-bb3f-e82ed43d3dc9","Type":"ContainerDied","Data":"e2132a8575a2ca963b909082cb3b2d867293ed4237b0cc980ec84c6995381ea6"}
Sep 11 21:43:23.653524 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:23.653160    2544 scope.go:117] "RemoveContainer" containerID="d58e006293c43c90469bed11522af8e82ad94d4849974bf82926658bd63d75c2"
Sep 11 21:43:23.662352 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:23.662326    2544 scope.go:117] "RemoveContainer" containerID="d58e006293c43c90469bed11522af8e82ad94d4849974bf82926658bd63d75c2"
Sep 11 21:43:23.662624 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:43:23.662604    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58e006293c43c90469bed11522af8e82ad94d4849974bf82926658bd63d75c2\": container with ID starting with d58e006293c43c90469bed11522af8e82ad94d4849974bf82926658bd63d75c2 not found: ID does not exist" containerID="d58e006293c43c90469bed11522af8e82ad94d4849974bf82926658bd63d75c2"
Sep 11 21:43:23.662676 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:23.662633    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58e006293c43c90469bed11522af8e82ad94d4849974bf82926658bd63d75c2"} err="failed to get container status \"d58e006293c43c90469bed11522af8e82ad94d4849974bf82926658bd63d75c2\": rpc error: code = NotFound desc = could not find container \"d58e006293c43c90469bed11522af8e82ad94d4849974bf82926658bd63d75c2\": container with ID starting with d58e006293c43c90469bed11522af8e82ad94d4849974bf82926658bd63d75c2 not found: ID does not exist"
Sep 11 21:43:23.670557 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:23.670535    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85cbf77b49-vqtz8"]
Sep 11 21:43:23.673804 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:23.673781    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-85cbf77b49-vqtz8"]
Sep 11 21:43:25.592956 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:25.592914    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65761e54-97e9-40bb-bb3f-e82ed43d3dc9" path="/var/lib/kubelet/pods/65761e54-97e9-40bb-bb3f-e82ed43d3dc9/volumes"
Sep 11 21:43:39.910925 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:39.910888    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"]
Sep 11 21:43:39.911397 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:39.911368    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65761e54-97e9-40bb-bb3f-e82ed43d3dc9" containerName="authorino"
Sep 11 21:43:39.911397 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:39.911387    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="65761e54-97e9-40bb-bb3f-e82ed43d3dc9" containerName="authorino"
Sep 11 21:43:39.911507 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:39.911469    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="65761e54-97e9-40bb-bb3f-e82ed43d3dc9" containerName="authorino"
Sep 11 21:43:39.915536 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:39.915514    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:39.918022 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:39.917990    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e919c5-openshift-default-dockercfg-kspvd\""
Sep 11 21:43:39.927168 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:39.927144    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"]
Sep 11 21:43:40.040302 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.040270    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"]
Sep 11 21:43:40.043867 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.043569    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.046649 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.046584    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-35e777-openshift-default-dockercfg-vf58k\""
Sep 11 21:43:40.057080 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.057057    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"]
Sep 11 21:43:40.091517 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.091477    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-workload-socket\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.091517 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.091522    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794b359e-9757-4166-aa5d-f45daae91981-istio-token\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.091752 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.091661    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhqbn\" (UniqueName: \"kubernetes.io/projected/794b359e-9757-4166-aa5d-f45daae91981-kube-api-access-dhqbn\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.091752 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.091717    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-credential-socket\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.091866 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.091766    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794b359e-9757-4166-aa5d-f45daae91981-istio-podinfo\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.091866 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.091804    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794b359e-9757-4166-aa5d-f45daae91981-istiod-ca-cert\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.091945 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.091882    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-istio-envoy\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.091945 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.091919    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-workload-certs\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.091945 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.091936    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-istio-data\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.193065 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.192972    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794b359e-9757-4166-aa5d-f45daae91981-istiod-ca-cert\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.193065 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193034    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-credential-socket\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.193065 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193059    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-data\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.193385 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193098    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-envoy\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.193385 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193147    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-istio-envoy\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.193385 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193195    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-token\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.193385 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193249    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-workload-certs\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.193385 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193268    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-istio-data\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.193385 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193290    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-workload-socket\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.193385 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193325    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-podinfo\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.193385 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193354    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794b359e-9757-4166-aa5d-f45daae91981-istio-token\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.193784 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193409    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istiod-ca-cert\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.193784 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193457    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-workload-certs\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.193784 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193492    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhqbn\" (UniqueName: \"kubernetes.io/projected/794b359e-9757-4166-aa5d-f45daae91981-kube-api-access-dhqbn\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.193784 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193519    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf2z9\" (UniqueName: \"kubernetes.io/projected/1a20e93f-ea55-42d2-9f62-1b971c171d7b-kube-api-access-gf2z9\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.193784 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193564    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-credential-socket\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.193784 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193589    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-workload-socket\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.193784 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193626    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-workload-certs\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.193784 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193669    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-istio-data\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.193784 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193763    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794b359e-9757-4166-aa5d-f45daae91981-istio-podinfo\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.194241 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193846    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794b359e-9757-4166-aa5d-f45daae91981-istiod-ca-cert\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.194241 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193873    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-credential-socket\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.194241 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.193921    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-workload-socket\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.196186 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.196154    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-istio-envoy\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.196664 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.196644    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794b359e-9757-4166-aa5d-f45daae91981-istio-podinfo\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.201724 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.201703    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhqbn\" (UniqueName: \"kubernetes.io/projected/794b359e-9757-4166-aa5d-f45daae91981-kube-api-access-dhqbn\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.201938 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.201917    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794b359e-9757-4166-aa5d-f45daae91981-istio-token\") pod \"e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") " pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.225999 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.225966    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:40.295131 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.295092    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-envoy\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.295310 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.295156    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-token\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.295310 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.295197    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-podinfo\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.295310 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.295273    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istiod-ca-cert\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.295476 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.295317    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-workload-certs\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.296812 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.296580    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-workload-certs\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.296946 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.296926    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istiod-ca-cert\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.297830 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.295350    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2z9\" (UniqueName: \"kubernetes.io/projected/1a20e93f-ea55-42d2-9f62-1b971c171d7b-kube-api-access-gf2z9\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.297830 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.297355    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-workload-socket\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.297830 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.297432    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-credential-socket\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.297830 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.297460    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-data\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.297830 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.297671    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-workload-socket\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.297830 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.297790    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-data\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.298235 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.297932    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-credential-socket\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.300377 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.300181    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-podinfo\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.301517 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.301118    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-envoy\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.304160 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.304125    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-token\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.304847 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.304827    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2z9\" (UniqueName: \"kubernetes.io/projected/1a20e93f-ea55-42d2-9f62-1b971c171d7b-kube-api-access-gf2z9\") pod \"e2e-worker-w4-35e777-openshift-default-775c996888-28smq\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") " pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.357657 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.357614    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:40.513840 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.513758    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"]
Sep 11 21:43:40.515287 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:43:40.515258    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794b359e_9757_4166_aa5d_f45daae91981.slice/crio-282ec0853e5687f77acc0a0f00bf074d0a6ac0d30da38373feb614e7729ea697 WatchSource:0}: Error finding container 282ec0853e5687f77acc0a0f00bf074d0a6ac0d30da38373feb614e7729ea697: Status 404 returned error can't find the container with id 282ec0853e5687f77acc0a0f00bf074d0a6ac0d30da38373feb614e7729ea697
Sep 11 21:43:40.517437 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.517406    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:43:40.517489 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.517471    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:43:40.517527 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.517498    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:43:40.632820 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.632788    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"]
Sep 11 21:43:40.634004 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:43:40.633977    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a20e93f_ea55_42d2_9f62_1b971c171d7b.slice/crio-8f275d83e69f8575e178522e181d9b93b4e83a630e8f7e108cd1a724e6527c58 WatchSource:0}: Error finding container 8f275d83e69f8575e178522e181d9b93b4e83a630e8f7e108cd1a724e6527c58: Status 404 returned error can't find the container with id 8f275d83e69f8575e178522e181d9b93b4e83a630e8f7e108cd1a724e6527c58
Sep 11 21:43:40.636503 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.636465    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:43:40.636587 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.636553    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:43:40.636627 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.636598    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:43:40.726853 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.726813    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq" event={"ID":"1a20e93f-ea55-42d2-9f62-1b971c171d7b","Type":"ContainerStarted","Data":"954d78cdd9743fddfdc7db61c96d687fb455bd73066f0fb7016f940e8c3db64c"}
Sep 11 21:43:40.727084 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.726878    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq" event={"ID":"1a20e93f-ea55-42d2-9f62-1b971c171d7b","Type":"ContainerStarted","Data":"8f275d83e69f8575e178522e181d9b93b4e83a630e8f7e108cd1a724e6527c58"}
Sep 11 21:43:40.729414 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.729383    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m" event={"ID":"794b359e-9757-4166-aa5d-f45daae91981","Type":"ContainerStarted","Data":"5c1ce71bbad49e45020db838b67cd835935c58e773ed1f715f5ed52753f98578"}
Sep 11 21:43:40.729541 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.729430    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m" event={"ID":"794b359e-9757-4166-aa5d-f45daae91981","Type":"ContainerStarted","Data":"282ec0853e5687f77acc0a0f00bf074d0a6ac0d30da38373feb614e7729ea697"}
Sep 11 21:43:40.747752 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.746613    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq" podStartSLOduration=0.746592732 podStartE2EDuration="746.592732ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:40.745995279 +0000 UTC m=+783.715412100" watchObservedRunningTime="2026-09-11 21:43:40.746592732 +0000 UTC m=+783.716009553"
Sep 11 21:43:40.767575 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:40.767465    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m" podStartSLOduration=1.767450314 podStartE2EDuration="1.767450314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:40.765608478 +0000 UTC m=+783.735025299" watchObservedRunningTime="2026-09-11 21:43:40.767450314 +0000 UTC m=+783.736867133"
Sep 11 21:43:41.227137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:41.227108    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:41.231946 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:41.231920    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:41.358056 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:41.358022    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:41.362938 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:41.362910    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:41.735148 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:41.735113    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:41.735343 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:41.735169    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:41.736050 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:41.736027    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:43:41.736198 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:41.736181    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:43:48.840905 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.840868    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"]
Sep 11 21:43:48.845561 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.845538    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.847992 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.847968    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-5b160c-openshift-default-dockercfg-rdb75\""
Sep 11 21:43:48.887656 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.887619    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-envoy\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.887821 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.887666    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-token\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.887821 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.887697    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-workload-socket\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.887821 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.887725    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-workload-certs\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.887821 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.887752    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-data\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.887821 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.887775    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a92534-26f9-425d-b6e5-756ba5c2d69f-istiod-ca-cert\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.888034 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.887823    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-podinfo\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.888034 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.887869    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpm7l\" (UniqueName: \"kubernetes.io/projected/33a92534-26f9-425d-b6e5-756ba5c2d69f-kube-api-access-hpm7l\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.888034 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.887919    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-credential-socket\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.911865 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.911831    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"]
Sep 11 21:43:48.928190 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.928156    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"]
Sep 11 21:43:48.931555 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.931529    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:48.934034 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.934011    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b2d1bd14-openshift-default-dockercfg-kw4k9\""
Sep 11 21:43:48.944904 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.944881    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"]
Sep 11 21:43:48.988433 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.988405    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-envoy\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.988617 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.988447    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-token\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.988617 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.988475    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-workload-socket\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.988617 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.988500    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-workload-certs\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.988763 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.988631    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-data\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.988763 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.988679    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a92534-26f9-425d-b6e5-756ba5c2d69f-istiod-ca-cert\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.988763 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.988717    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-credential-socket\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:48.988937 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.988799    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-podinfo\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.988937 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.988826    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpm7l\" (UniqueName: \"kubernetes.io/projected/33a92534-26f9-425d-b6e5-756ba5c2d69f-kube-api-access-hpm7l\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.988937 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.988857    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-workload-certs\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:48.988937 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.988881    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-istio-data\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:48.988937 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.988914    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-workload-certs\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.988937 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.988928    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a71664-0437-47fb-98a2-660db9026270-istio-token\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:48.989266 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.988952    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a71664-0437-47fb-98a2-660db9026270-istiod-ca-cert\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:48.989266 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.988956    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-workload-socket\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.989266 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.989005    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-credential-socket\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.989266 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.989084    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-workload-socket\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:48.989266 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.989149    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-istio-envoy\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:48.989266 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.989178    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a71664-0437-47fb-98a2-660db9026270-istio-podinfo\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:48.989580 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.989269    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sldp\" (UniqueName: \"kubernetes.io/projected/17a71664-0437-47fb-98a2-660db9026270-kube-api-access-4sldp\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:48.989580 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.989368    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-credential-socket\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.989580 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.989461    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-data\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.989580 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.989465    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a92534-26f9-425d-b6e5-756ba5c2d69f-istiod-ca-cert\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.991066 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.991044    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-envoy\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.991335 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.991317    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-podinfo\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.997264 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.997234    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-token\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:48.997870 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:48.997846    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpm7l\" (UniqueName: \"kubernetes.io/projected/33a92534-26f9-425d-b6e5-756ba5c2d69f-kube-api-access-hpm7l\") pod \"e2e-autores-5b160c-openshift-default-6788979f8d-tv587\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") " pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:49.090685 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.090648    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-workload-certs\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.090876 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.090693    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-istio-data\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.090876 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.090740    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a71664-0437-47fb-98a2-660db9026270-istio-token\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.090876 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.090766    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a71664-0437-47fb-98a2-660db9026270-istiod-ca-cert\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.090876 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.090798    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-workload-socket\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.090876 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.090840    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-istio-envoy\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.090876 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.090865    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a71664-0437-47fb-98a2-660db9026270-istio-podinfo\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.091209 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.090896    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sldp\" (UniqueName: \"kubernetes.io/projected/17a71664-0437-47fb-98a2-660db9026270-kube-api-access-4sldp\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.091209 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.090999    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-credential-socket\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.091209 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.091149    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-workload-certs\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.091748 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.091415    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-istio-data\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.091855 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.091814    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-credential-socket\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.092565 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.092541    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a71664-0437-47fb-98a2-660db9026270-istiod-ca-cert\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.092690 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.092639    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-workload-socket\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.094111 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.094024    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-istio-envoy\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.094804 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.094777    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a71664-0437-47fb-98a2-660db9026270-istio-podinfo\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.101996 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.101972    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a71664-0437-47fb-98a2-660db9026270-istio-token\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.102084 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.102021    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sldp\" (UniqueName: \"kubernetes.io/projected/17a71664-0437-47fb-98a2-660db9026270-kube-api-access-4sldp\") pod \"e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") " pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.159233 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.159184    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:49.242125 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.242089    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:49.433526 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.433499    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"]
Sep 11 21:43:49.435208 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:43:49.435187    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a92534_26f9_425d_b6e5_756ba5c2d69f.slice/crio-0acaba823ce3d7d2425abbc6861f0bdc867f407fa0c6fdbc43d121fd13f4a328 WatchSource:0}: Error finding container 0acaba823ce3d7d2425abbc6861f0bdc867f407fa0c6fdbc43d121fd13f4a328: Status 404 returned error can't find the container with id 0acaba823ce3d7d2425abbc6861f0bdc867f407fa0c6fdbc43d121fd13f4a328
Sep 11 21:43:49.437399 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.437365    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:43:49.437507 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.437439    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:43:49.437507 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.437482    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:43:49.518041 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.518010    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"]
Sep 11 21:43:49.519493 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:43:49.519459    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a71664_0437_47fb_98a2_660db9026270.slice/crio-f8a98ccc1c019d976fffc235ce7a393bca7df9e3624848bddcce78622f328bf7 WatchSource:0}: Error finding container f8a98ccc1c019d976fffc235ce7a393bca7df9e3624848bddcce78622f328bf7: Status 404 returned error can't find the container with id f8a98ccc1c019d976fffc235ce7a393bca7df9e3624848bddcce78622f328bf7
Sep 11 21:43:49.521765 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.521735    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:43:49.521830 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.521803    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:43:49.521867 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.521834    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:43:49.771207 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.771168    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj" event={"ID":"17a71664-0437-47fb-98a2-660db9026270","Type":"ContainerStarted","Data":"11badb6589f2b145c96c6819ecfcfd8f8d687cea962f01e838c127421fff9d5e"}
Sep 11 21:43:49.771417 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.771256    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj" event={"ID":"17a71664-0437-47fb-98a2-660db9026270","Type":"ContainerStarted","Data":"f8a98ccc1c019d976fffc235ce7a393bca7df9e3624848bddcce78622f328bf7"}
Sep 11 21:43:49.773169 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.773127    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587" event={"ID":"33a92534-26f9-425d-b6e5-756ba5c2d69f","Type":"ContainerStarted","Data":"b0c0843834c12de2c5b195d9b557e4e28d4bba63b27fdc76e1872553c6df9fff"}
Sep 11 21:43:49.773313 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.773180    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587" event={"ID":"33a92534-26f9-425d-b6e5-756ba5c2d69f","Type":"ContainerStarted","Data":"0acaba823ce3d7d2425abbc6861f0bdc867f407fa0c6fdbc43d121fd13f4a328"}
Sep 11 21:43:49.792420 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.792361    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj" podStartSLOduration=1.792340501 podStartE2EDuration="1.792340501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:49.790817852 +0000 UTC m=+792.760234672" watchObservedRunningTime="2026-09-11 21:43:49.792340501 +0000 UTC m=+792.761757322"
Sep 11 21:43:49.810675 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:49.810569    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587" podStartSLOduration=1.810548316 podStartE2EDuration="1.810548316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:49.809594193 +0000 UTC m=+792.779011037" watchObservedRunningTime="2026-09-11 21:43:49.810548316 +0000 UTC m=+792.779965140"
Sep 11 21:43:50.159731 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:50.159645    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:50.164395 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:50.164371    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:50.242522 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:50.242478    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:50.247557 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:50.247533    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:50.777691 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:50.777652    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:50.777920 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:50.777901    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:43:50.778469 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:50.778446    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:43:50.778593 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:43:50.778575    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:44:02.837874 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:02.837839    2544 generic.go:356] "Generic (PLEG): container finished" podID="1a20e93f-ea55-42d2-9f62-1b971c171d7b" containerID="954d78cdd9743fddfdc7db61c96d687fb455bd73066f0fb7016f940e8c3db64c" exitCode=0
Sep 11 21:44:02.838362 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:02.837914    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq" event={"ID":"1a20e93f-ea55-42d2-9f62-1b971c171d7b","Type":"ContainerDied","Data":"954d78cdd9743fddfdc7db61c96d687fb455bd73066f0fb7016f940e8c3db64c"}
Sep 11 21:44:02.838453 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:02.838423    2544 scope.go:117] "RemoveContainer" containerID="954d78cdd9743fddfdc7db61c96d687fb455bd73066f0fb7016f940e8c3db64c"
Sep 11 21:44:02.839623 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:02.839601    2544 generic.go:356] "Generic (PLEG): container finished" podID="794b359e-9757-4166-aa5d-f45daae91981" containerID="5c1ce71bbad49e45020db838b67cd835935c58e773ed1f715f5ed52753f98578" exitCode=0
Sep 11 21:44:02.839721 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:02.839648    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m" event={"ID":"794b359e-9757-4166-aa5d-f45daae91981","Type":"ContainerDied","Data":"5c1ce71bbad49e45020db838b67cd835935c58e773ed1f715f5ed52753f98578"}
Sep 11 21:44:02.840265 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:02.840204    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:02.840416 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:02.840394    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:02.840570 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:02.840552    2544 scope.go:117] "RemoveContainer" containerID="5c1ce71bbad49e45020db838b67cd835935c58e773ed1f715f5ed52753f98578"
Sep 11 21:44:02.840627 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:02.840565    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:02.844718 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:02.843603    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:02.844718 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:02.843736    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:02.844718 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:02.843827    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:03.849816 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:03.849771    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m" event={"ID":"794b359e-9757-4166-aa5d-f45daae91981","Type":"ContainerStarted","Data":"6fb52e4bee67ccd22dcef7009a6b7f5065f47214c36456912c865563981aa4c7"}
Sep 11 21:44:03.852828 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:03.852792    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq" event={"ID":"1a20e93f-ea55-42d2-9f62-1b971c171d7b","Type":"ContainerStarted","Data":"d681eac6b494106c20830ed9966ad29a39465546730bcc9de48e7d2be3f77dc6"}
Sep 11 21:44:04.227008 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:04.226969    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:44:04.227444 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:04.227414    2544 patch_prober.go:28] interesting pod/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 11 21:44:04.227547 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:04.227475    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m" podUID="794b359e-9757-4166-aa5d-f45daae91981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 11 21:44:04.358556 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:04.358520    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:44:04.358980 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:04.358864    2544 patch_prober.go:28] interesting pod/e2e-worker-w4-35e777-openshift-default-775c996888-28smq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 11 21:44:04.359157 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:04.359134    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq" podUID="1a20e93f-ea55-42d2-9f62-1b971c171d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 11 21:44:05.227126 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:05.227083    2544 patch_prober.go:28] interesting pod/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 11 21:44:05.227625 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:05.227160    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m" podUID="794b359e-9757-4166-aa5d-f45daae91981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 11 21:44:05.358767 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:05.358731    2544 patch_prober.go:28] interesting pod/e2e-worker-w4-35e777-openshift-default-775c996888-28smq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 11 21:44:05.359086 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:05.359035    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq" podUID="1a20e93f-ea55-42d2-9f62-1b971c171d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 11 21:44:06.226819 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:06.226778    2544 patch_prober.go:28] interesting pod/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 11 21:44:06.227118 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:06.227086    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m" podUID="794b359e-9757-4166-aa5d-f45daae91981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 11 21:44:06.359093 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:06.359043    2544 patch_prober.go:28] interesting pod/e2e-worker-w4-35e777-openshift-default-775c996888-28smq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 11 21:44:06.359626 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:06.359130    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq" podUID="1a20e93f-ea55-42d2-9f62-1b971c171d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 11 21:44:07.227643 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:07.227609    2544 patch_prober.go:28] interesting pod/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 11 21:44:07.227917 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:07.227891    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m" podUID="794b359e-9757-4166-aa5d-f45daae91981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 11 21:44:07.358485 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:07.358448    2544 patch_prober.go:28] interesting pod/e2e-worker-w4-35e777-openshift-default-775c996888-28smq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 11 21:44:07.358805 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:07.358758    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq" podUID="1a20e93f-ea55-42d2-9f62-1b971c171d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 11 21:44:08.226623 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.226582    2544 patch_prober.go:28] interesting pod/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 11 21:44:08.227204 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.227168    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m" podUID="794b359e-9757-4166-aa5d-f45daae91981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 11 21:44:08.359775 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.359731    2544 patch_prober.go:28] interesting pod/e2e-worker-w4-35e777-openshift-default-775c996888-28smq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 11 21:44:08.359981 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.359816    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq" podUID="1a20e93f-ea55-42d2-9f62-1b971c171d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 11 21:44:08.449549 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.449517    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"]
Sep 11 21:44:08.458140 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.458114    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.463411 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.463383    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7e6f5e-openshift-default-dockercfg-jj857\""
Sep 11 21:44:08.486686 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.486610    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"]
Sep 11 21:44:08.622868 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.622823    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-credential-socket\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.623146 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.623124    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-token\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.623418 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.623387    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-envoy\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.623580 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.623559    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istiod-ca-cert\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.623727 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.623711    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-podinfo\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.623873 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.623856    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88m5d\" (UniqueName: \"kubernetes.io/projected/f597fc9d-2f8e-4047-970b-2d85ca4a928d-kube-api-access-88m5d\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.624007 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.623992    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-data\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.624143 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.624128    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-workload-socket\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.624275 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.624260    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-workload-certs\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.726061 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.726025    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-podinfo\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.726411 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.726372    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88m5d\" (UniqueName: \"kubernetes.io/projected/f597fc9d-2f8e-4047-970b-2d85ca4a928d-kube-api-access-88m5d\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.726560 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.726543    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-data\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.726696 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.726681    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-workload-socket\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.726814 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.726801    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-workload-certs\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.726989 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.726975    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-credential-socket\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.727113 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.727099    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-token\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.727371 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.727355    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-envoy\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.727512 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.727494    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istiod-ca-cert\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.728612 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.728572    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istiod-ca-cert\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.729632 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.729610    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-workload-certs\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.732247 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.730353    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-credential-socket\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.732392 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.730885    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-data\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.732583 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.731397    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-workload-socket\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.732701 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.732159    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-podinfo\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.737072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.737011    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-envoy\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.741421 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.741399    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88m5d\" (UniqueName: \"kubernetes.io/projected/f597fc9d-2f8e-4047-970b-2d85ca4a928d-kube-api-access-88m5d\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.742728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.742707    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-token\") pod \"e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") " pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:08.795961 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:08.795912    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:09.228766 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:09.228721    2544 patch_prober.go:28] interesting pod/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 11 21:44:09.229268 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:09.228806    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m" podUID="794b359e-9757-4166-aa5d-f45daae91981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 11 21:44:09.359164 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:09.358711    2544 patch_prober.go:28] interesting pod/e2e-worker-w4-35e777-openshift-default-775c996888-28smq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 11 21:44:09.359164 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:09.358778    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq" podUID="1a20e93f-ea55-42d2-9f62-1b971c171d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 11 21:44:09.394676 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:09.394479    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"]
Sep 11 21:44:09.411296 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:44:09.411255    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf597fc9d_2f8e_4047_970b_2d85ca4a928d.slice/crio-2c990eabeb84fb41c9cdfc2ffb6f140a40142605566c993cd08170c8357f838f WatchSource:0}: Error finding container 2c990eabeb84fb41c9cdfc2ffb6f140a40142605566c993cd08170c8357f838f: Status 404 returned error can't find the container with id 2c990eabeb84fb41c9cdfc2ffb6f140a40142605566c993cd08170c8357f838f
Sep 11 21:44:09.415791 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:09.415744    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:09.416017 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:09.415995    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:09.416234 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:09.416152    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:09.909578 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:09.909531    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9" event={"ID":"f597fc9d-2f8e-4047-970b-2d85ca4a928d","Type":"ContainerStarted","Data":"8622ed6d9b23ae897278a035a57464c8b84a341294ac4b19007db1af147a0607"}
Sep 11 21:44:09.909794 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:09.909638    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9" event={"ID":"f597fc9d-2f8e-4047-970b-2d85ca4a928d","Type":"ContainerStarted","Data":"2c990eabeb84fb41c9cdfc2ffb6f140a40142605566c993cd08170c8357f838f"}
Sep 11 21:44:09.941513 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:09.941449    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9" podStartSLOduration=1.9414282059999999 podStartE2EDuration="1.941428206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:44:09.936666516 +0000 UTC m=+812.906083338" watchObservedRunningTime="2026-09-11 21:44:09.941428206 +0000 UTC m=+812.910845039"
Sep 11 21:44:10.364952 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:10.364901    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:44:10.365441 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:10.365420    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:44:10.388295 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:10.387477    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:44:10.731894 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:10.731854    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:44:10.733159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:10.733119    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:44:10.738452 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:10.738428    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:44:10.796984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:10.796942    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:10.805260 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:10.805239    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:10.914080 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:10.914053    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:10.914700 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:10.914682    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:11.555337 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:11.555299    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2d1bd14-6gxhp"]
Sep 11 21:44:11.558969 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:11.558939    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2d1bd14-6gxhp"
Sep 11 21:44:11.561559 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:11.561521    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 21:44:11.561873 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:11.561850    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:44:11.570425 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:11.570387    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2d1bd14-6gxhp"]
Sep 11 21:44:11.587333 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:11.587306    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdcb836e-8a14-430f-9ff5-50f5d7ec04fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b2d1bd14-6gxhp\" (UID: \"bdcb836e-8a14-430f-9ff5-50f5d7ec04fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2d1bd14-6gxhp"
Sep 11 21:44:11.688707 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:11.688665    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdcb836e-8a14-430f-9ff5-50f5d7ec04fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b2d1bd14-6gxhp\" (UID: \"bdcb836e-8a14-430f-9ff5-50f5d7ec04fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2d1bd14-6gxhp"
Sep 11 21:44:11.689366 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:11.689348    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdcb836e-8a14-430f-9ff5-50f5d7ec04fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b2d1bd14-6gxhp\" (UID: \"bdcb836e-8a14-430f-9ff5-50f5d7ec04fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2d1bd14-6gxhp"
Sep 11 21:44:11.880544 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:11.880453    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2d1bd14-6gxhp"
Sep 11 21:44:12.199741 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:12.199718    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2d1bd14-6gxhp"]
Sep 11 21:44:12.201690 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:44:12.201662    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdcb836e_8a14_430f_9ff5_50f5d7ec04fb.slice/crio-57ac2de3dfbfd8883ccf73fc91b99226316151ebf0f29fb6b65e029f557ddbb8 WatchSource:0}: Error finding container 57ac2de3dfbfd8883ccf73fc91b99226316151ebf0f29fb6b65e029f557ddbb8: Status 404 returned error can't find the container with id 57ac2de3dfbfd8883ccf73fc91b99226316151ebf0f29fb6b65e029f557ddbb8
Sep 11 21:44:12.926173 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:12.926126    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2d1bd14-6gxhp" event={"ID":"bdcb836e-8a14-430f-9ff5-50f5d7ec04fb","Type":"ContainerStarted","Data":"57ac2de3dfbfd8883ccf73fc91b99226316151ebf0f29fb6b65e029f557ddbb8"}
Sep 11 21:44:15.947255 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:15.947138    2544 generic.go:356] "Generic (PLEG): container finished" podID="bdcb836e-8a14-430f-9ff5-50f5d7ec04fb" containerID="a1b884f5f4561f599b4cd1a393f68c3a18f3955a9d54bce0c880f931e481377d" exitCode=0
Sep 11 21:44:15.947642 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:15.947256    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2d1bd14-6gxhp" event={"ID":"bdcb836e-8a14-430f-9ff5-50f5d7ec04fb","Type":"ContainerDied","Data":"a1b884f5f4561f599b4cd1a393f68c3a18f3955a9d54bce0c880f931e481377d"}
Sep 11 21:44:16.467858 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.467822    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf"]
Sep 11 21:44:16.469970 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.469954    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf"
Sep 11 21:44:16.473263 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.473237    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7e6f5e\""
Sep 11 21:44:16.476904 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.476881    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7e6f5e-dockercfg-5fctf\""
Sep 11 21:44:16.490934 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.490907    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf"]
Sep 11 21:44:16.542364 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.542327    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf7kg\" (UniqueName: \"kubernetes.io/projected/3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908-kube-api-access-xf7kg\") pod \"payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf\" (UID: \"3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf"
Sep 11 21:44:16.542499 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.542477    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf\" (UID: \"3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf"
Sep 11 21:44:16.643849 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.643794    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf7kg\" (UniqueName: \"kubernetes.io/projected/3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908-kube-api-access-xf7kg\") pod \"payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf\" (UID: \"3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf"
Sep 11 21:44:16.644117 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.644087    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf\" (UID: \"3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf"
Sep 11 21:44:16.644827 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.644803    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf\" (UID: \"3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf"
Sep 11 21:44:16.651923 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.651896    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf7kg\" (UniqueName: \"kubernetes.io/projected/3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908-kube-api-access-xf7kg\") pod \"payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf\" (UID: \"3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf"
Sep 11 21:44:16.780105 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.780014    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf"
Sep 11 21:44:16.954397 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.954365    2544 generic.go:356] "Generic (PLEG): container finished" podID="33a92534-26f9-425d-b6e5-756ba5c2d69f" containerID="b0c0843834c12de2c5b195d9b557e4e28d4bba63b27fdc76e1872553c6df9fff" exitCode=0
Sep 11 21:44:16.954821 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.954444    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587" event={"ID":"33a92534-26f9-425d-b6e5-756ba5c2d69f","Type":"ContainerDied","Data":"b0c0843834c12de2c5b195d9b557e4e28d4bba63b27fdc76e1872553c6df9fff"}
Sep 11 21:44:16.954916 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.954894    2544 scope.go:117] "RemoveContainer" containerID="b0c0843834c12de2c5b195d9b557e4e28d4bba63b27fdc76e1872553c6df9fff"
Sep 11 21:44:16.956151 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.956122    2544 generic.go:356] "Generic (PLEG): container finished" podID="17a71664-0437-47fb-98a2-660db9026270" containerID="11badb6589f2b145c96c6819ecfcfd8f8d687cea962f01e838c127421fff9d5e" exitCode=0
Sep 11 21:44:16.956244 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.956149    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:16.956244 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.956169    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj" event={"ID":"17a71664-0437-47fb-98a2-660db9026270","Type":"ContainerDied","Data":"11badb6589f2b145c96c6819ecfcfd8f8d687cea962f01e838c127421fff9d5e"}
Sep 11 21:44:16.956244 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.956207    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:16.956355 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.956261    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:16.956649 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.956635    2544 scope.go:117] "RemoveContainer" containerID="11badb6589f2b145c96c6819ecfcfd8f8d687cea962f01e838c127421fff9d5e"
Sep 11 21:44:16.957781 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.957755    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:16.957882 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.957855    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:16.960835 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:16.958431    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:17.051935 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:17.051899    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf"]
Sep 11 21:44:17.121601 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:17.121579    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2d1bd14-6gxhp"
Sep 11 21:44:17.251675 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:17.251640    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdcb836e-8a14-430f-9ff5-50f5d7ec04fb-maas-api-service-ca\") pod \"bdcb836e-8a14-430f-9ff5-50f5d7ec04fb\" (UID: \"bdcb836e-8a14-430f-9ff5-50f5d7ec04fb\") "
Sep 11 21:44:17.252071 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:17.252041    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdcb836e-8a14-430f-9ff5-50f5d7ec04fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bdcb836e-8a14-430f-9ff5-50f5d7ec04fb" (UID: "bdcb836e-8a14-430f-9ff5-50f5d7ec04fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:44:17.353174 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:17.353094    2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdcb836e-8a14-430f-9ff5-50f5d7ec04fb-maas-api-service-ca\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:44:17.974249 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:17.973821    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587" event={"ID":"33a92534-26f9-425d-b6e5-756ba5c2d69f","Type":"ContainerStarted","Data":"a01f8a8e5f8d61626c2aa2b21a080871bf7228b99433a89850319fb8e773ff7a"}
Sep 11 21:44:17.977601 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:17.977517    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf" event={"ID":"3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908","Type":"ContainerStarted","Data":"5f609eac6ff9e7e20127c8a6858f0c63e5241fa0cfc82a5642b6a4eae0a4a329"}
Sep 11 21:44:17.981910 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:17.981186    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj" event={"ID":"17a71664-0437-47fb-98a2-660db9026270","Type":"ContainerStarted","Data":"8ecb04f8fe991e94403dd37500469e55a1b6cc6fd342933c0196f7732fd06c9f"}
Sep 11 21:44:17.992692 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:17.992655    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2d1bd14-6gxhp" event={"ID":"bdcb836e-8a14-430f-9ff5-50f5d7ec04fb","Type":"ContainerDied","Data":"57ac2de3dfbfd8883ccf73fc91b99226316151ebf0f29fb6b65e029f557ddbb8"}
Sep 11 21:44:17.992873 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:17.992724    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57ac2de3dfbfd8883ccf73fc91b99226316151ebf0f29fb6b65e029f557ddbb8"
Sep 11 21:44:17.992873 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:17.992673    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2d1bd14-6gxhp"
Sep 11 21:44:18.161696 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:18.161604    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:44:18.173244 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:18.168656    2544 patch_prober.go:28] interesting pod/e2e-autores-5b160c-openshift-default-6788979f8d-tv587 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 11 21:44:18.173244 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:18.168721    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587" podUID="33a92534-26f9-425d-b6e5-756ba5c2d69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 11 21:44:18.251578 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:18.251456    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:44:18.251752 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:18.251633    2544 patch_prober.go:28] interesting pod/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 11 21:44:18.251752 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:18.251675    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj" podUID="17a71664-0437-47fb-98a2-660db9026270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 11 21:44:19.163144 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:19.162674    2544 patch_prober.go:28] interesting pod/e2e-autores-5b160c-openshift-default-6788979f8d-tv587 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 11 21:44:19.163144 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:19.162748    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587" podUID="33a92534-26f9-425d-b6e5-756ba5c2d69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 11 21:44:19.244088 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:19.243635    2544 patch_prober.go:28] interesting pod/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 11 21:44:19.244088 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:19.243705    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj" podUID="17a71664-0437-47fb-98a2-660db9026270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 11 21:44:20.160245 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:20.159753    2544 patch_prober.go:28] interesting pod/e2e-autores-5b160c-openshift-default-6788979f8d-tv587 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 11 21:44:20.160245 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:20.159821    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587" podUID="33a92534-26f9-425d-b6e5-756ba5c2d69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 11 21:44:20.253710 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:20.253663    2544 patch_prober.go:28] interesting pod/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 11 21:44:20.254163 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:20.253733    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj" podUID="17a71664-0437-47fb-98a2-660db9026270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 11 21:44:20.778385 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:20.778328    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:44:20.778599 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:20.778436    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:44:21.160054 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:21.159602    2544 patch_prober.go:28] interesting pod/e2e-autores-5b160c-openshift-default-6788979f8d-tv587 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 11 21:44:21.160054 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:21.159669    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587" podUID="33a92534-26f9-425d-b6e5-756ba5c2d69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 11 21:44:21.243669 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:21.243184    2544 patch_prober.go:28] interesting pod/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 11 21:44:21.243669 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:21.243275    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj" podUID="17a71664-0437-47fb-98a2-660db9026270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 11 21:44:22.013443 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:22.013405    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf" event={"ID":"3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908","Type":"ContainerStarted","Data":"72fcfd91c28b90c5176e25b2eba8822f6f659a0ae73b82335440fcfba62eb8a9"}
Sep 11 21:44:22.013848 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:22.013473    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf"
Sep 11 21:44:22.036676 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:22.036582    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf" podStartSLOduration=2.052481718 podStartE2EDuration="6.036540614s" totalImagesPullingTime="3.984058896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:44:22.03203687 +0000 UTC m=+825.001453690" watchObservedRunningTime="2026-09-11 21:44:22.036540614 +0000 UTC m=+825.005957435"
Sep 11 21:44:22.161187 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:22.160728    2544 patch_prober.go:28] interesting pod/e2e-autores-5b160c-openshift-default-6788979f8d-tv587 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 11 21:44:22.161187 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:22.160796    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587" podUID="33a92534-26f9-425d-b6e5-756ba5c2d69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 11 21:44:22.243296 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:22.242822    2544 patch_prober.go:28] interesting pod/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 11 21:44:22.243296 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:22.242891    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj" podUID="17a71664-0437-47fb-98a2-660db9026270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 11 21:44:22.812307 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:22.812265    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2d1bd14-6gxhp"]
Sep 11 21:44:22.815683 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:22.815643    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2d1bd14-6gxhp"]
Sep 11 21:44:22.968411 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:22.968374    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"]
Sep 11 21:44:22.968893 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:22.968858    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj" podUID="17a71664-0437-47fb-98a2-660db9026270" containerName="istio-proxy" containerID="cri-o://8ecb04f8fe991e94403dd37500469e55a1b6cc6fd342933c0196f7732fd06c9f" gracePeriod=30
Sep 11 21:44:23.160510 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:23.160413    2544 patch_prober.go:28] interesting pod/e2e-autores-5b160c-openshift-default-6788979f8d-tv587 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 11 21:44:23.160510 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:23.160483    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587" podUID="33a92534-26f9-425d-b6e5-756ba5c2d69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 11 21:44:23.592573 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:23.592540    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdcb836e-8a14-430f-9ff5-50f5d7ec04fb" path="/var/lib/kubelet/pods/bdcb836e-8a14-430f-9ff5-50f5d7ec04fb/volumes"
Sep 11 21:44:23.949955 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:23.949877    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"]
Sep 11 21:44:23.950300 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:23.950280    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdcb836e-8a14-430f-9ff5-50f5d7ec04fb" containerName="revoke-keys"
Sep 11 21:44:23.950376 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:23.950301    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdcb836e-8a14-430f-9ff5-50f5d7ec04fb" containerName="revoke-keys"
Sep 11 21:44:23.950376 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:23.950363    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdcb836e-8a14-430f-9ff5-50f5d7ec04fb" containerName="revoke-keys"
Sep 11 21:44:23.955592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:23.955568    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:23.958215 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:23.958187    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-762b41ed-gw-openshift-default-dockercfg-th8z8\""
Sep 11 21:44:23.965037 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:23.965015    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"]
Sep 11 21:44:24.031759 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.031720    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-podinfo\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.031945 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.031787    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btnl2\" (UniqueName: \"kubernetes.io/projected/e9bf8917-07c3-44a2-bd90-f39662474b48-kube-api-access-btnl2\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.031945 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.031870    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bf8917-07c3-44a2-bd90-f39662474b48-istiod-ca-cert\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.031945 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.031920    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-data\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.032055 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.031982    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-workload-socket\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.032055 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.032021    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-credential-socket\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.032055 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.032048    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-token\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.032196 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.032176    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-envoy\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.032285 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.032246    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-workload-certs\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.133455 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.133412    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-credential-socket\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.133758 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.133726    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-token\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.133868 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.133841    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-envoy\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.133939 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.133871    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-workload-certs\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.133939 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.133881    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-credential-socket\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.133939 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.133909    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-podinfo\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.134106 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.133956    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btnl2\" (UniqueName: \"kubernetes.io/projected/e9bf8917-07c3-44a2-bd90-f39662474b48-kube-api-access-btnl2\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.134106 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.133988    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bf8917-07c3-44a2-bd90-f39662474b48-istiod-ca-cert\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.134106 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.134018    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-data\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.134106 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.134083    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-workload-socket\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.136509 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.134832    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-workload-socket\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.136509 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.135140    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-workload-certs\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.136509 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.135944    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bf8917-07c3-44a2-bd90-f39662474b48-istiod-ca-cert\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.136509 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.136458    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-data\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.150724 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.150679    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-podinfo\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.152511 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.152481    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-envoy\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.153707 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.153681    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-token\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.154804 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.154782    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btnl2\" (UniqueName: \"kubernetes.io/projected/e9bf8917-07c3-44a2-bd90-f39662474b48-kube-api-access-btnl2\") pod \"e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") " pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.164509 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.164484    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:44:24.165781 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.165754    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:44:24.266715 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.266680    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:24.549950 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.549924    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"]
Sep 11 21:44:24.551502 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:44:24.551467    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9bf8917_07c3_44a2_bd90_f39662474b48.slice/crio-42e31542a0d05fdc5c76dd1128cfdbed4c33115db9d0b4b7b8069b5093adc4ad WatchSource:0}: Error finding container 42e31542a0d05fdc5c76dd1128cfdbed4c33115db9d0b4b7b8069b5093adc4ad: Status 404 returned error can't find the container with id 42e31542a0d05fdc5c76dd1128cfdbed4c33115db9d0b4b7b8069b5093adc4ad
Sep 11 21:44:24.553815 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.553780    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:24.553911 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.553848    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:24.553911 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:24.553878    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:25.036503 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:25.036463    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285" event={"ID":"e9bf8917-07c3-44a2-bd90-f39662474b48","Type":"ContainerStarted","Data":"22d10f024d5a1ca8ede79a5308c73e54593a001c0a25640972a87685f3082bf9"}
Sep 11 21:44:25.036720 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:25.036525    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285" event={"ID":"e9bf8917-07c3-44a2-bd90-f39662474b48","Type":"ContainerStarted","Data":"42e31542a0d05fdc5c76dd1128cfdbed4c33115db9d0b4b7b8069b5093adc4ad"}
Sep 11 21:44:25.058809 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:25.058755    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285" podStartSLOduration=2.058739541 podStartE2EDuration="2.058739541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:44:25.057428402 +0000 UTC m=+828.026845224" watchObservedRunningTime="2026-09-11 21:44:25.058739541 +0000 UTC m=+828.028156360"
Sep 11 21:44:25.267511 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:25.267473    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:25.272878 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:25.272848    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:26.042664 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:26.042515    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:26.047066 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:26.047034    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:28.973103 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.973083    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:44:28.995753 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.995720    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-istio-data\") pod \"17a71664-0437-47fb-98a2-660db9026270\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") "
Sep 11 21:44:28.995906 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.995777    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-workload-certs\") pod \"17a71664-0437-47fb-98a2-660db9026270\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") "
Sep 11 21:44:28.995906 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.995815    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-istio-envoy\") pod \"17a71664-0437-47fb-98a2-660db9026270\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") "
Sep 11 21:44:28.995906 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.995841    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sldp\" (UniqueName: \"kubernetes.io/projected/17a71664-0437-47fb-98a2-660db9026270-kube-api-access-4sldp\") pod \"17a71664-0437-47fb-98a2-660db9026270\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") "
Sep 11 21:44:28.995906 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.995900    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a71664-0437-47fb-98a2-660db9026270-istiod-ca-cert\") pod \"17a71664-0437-47fb-98a2-660db9026270\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") "
Sep 11 21:44:28.996117 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.995939    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a71664-0437-47fb-98a2-660db9026270-istio-token\") pod \"17a71664-0437-47fb-98a2-660db9026270\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") "
Sep 11 21:44:28.996117 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.996008    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a71664-0437-47fb-98a2-660db9026270-istio-podinfo\") pod \"17a71664-0437-47fb-98a2-660db9026270\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") "
Sep 11 21:44:28.996117 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.996036    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-istio-data" (OuterVolumeSpecName: "istio-data") pod "17a71664-0437-47fb-98a2-660db9026270" (UID: "17a71664-0437-47fb-98a2-660db9026270"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:44:28.996117 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.996054    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "17a71664-0437-47fb-98a2-660db9026270" (UID: "17a71664-0437-47fb-98a2-660db9026270"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:44:28.996117 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.996044    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-workload-socket\") pod \"17a71664-0437-47fb-98a2-660db9026270\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") "
Sep 11 21:44:28.996375 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.996134    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-credential-socket\") pod \"17a71664-0437-47fb-98a2-660db9026270\" (UID: \"17a71664-0437-47fb-98a2-660db9026270\") "
Sep 11 21:44:28.996375 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.996300    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "17a71664-0437-47fb-98a2-660db9026270" (UID: "17a71664-0437-47fb-98a2-660db9026270"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:44:28.996906 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.996873    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a71664-0437-47fb-98a2-660db9026270-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "17a71664-0437-47fb-98a2-660db9026270" (UID: "17a71664-0437-47fb-98a2-660db9026270"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:44:28.997234 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.997196    2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-istio-data\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:44:28.997334 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.997242    2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-workload-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:44:28.997334 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.997265    2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-workload-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:44:28.998763 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.998737    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "17a71664-0437-47fb-98a2-660db9026270" (UID: "17a71664-0437-47fb-98a2-660db9026270"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:44:29.000178 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:28.999634    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "17a71664-0437-47fb-98a2-660db9026270" (UID: "17a71664-0437-47fb-98a2-660db9026270"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:44:29.007093 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.001756    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a71664-0437-47fb-98a2-660db9026270-kube-api-access-4sldp" (OuterVolumeSpecName: "kube-api-access-4sldp") pod "17a71664-0437-47fb-98a2-660db9026270" (UID: "17a71664-0437-47fb-98a2-660db9026270"). InnerVolumeSpecName "kube-api-access-4sldp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:44:29.007251 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.007143    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17a71664-0437-47fb-98a2-660db9026270-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "17a71664-0437-47fb-98a2-660db9026270" (UID: "17a71664-0437-47fb-98a2-660db9026270"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:44:29.007809 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.007652    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a71664-0437-47fb-98a2-660db9026270-istio-token" (OuterVolumeSpecName: "istio-token") pod "17a71664-0437-47fb-98a2-660db9026270" (UID: "17a71664-0437-47fb-98a2-660db9026270"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:44:29.059973 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.059933    2544 generic.go:356] "Generic (PLEG): container finished" podID="17a71664-0437-47fb-98a2-660db9026270" containerID="8ecb04f8fe991e94403dd37500469e55a1b6cc6fd342933c0196f7732fd06c9f" exitCode=0
Sep 11 21:44:29.060153 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.060033    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj" event={"ID":"17a71664-0437-47fb-98a2-660db9026270","Type":"ContainerDied","Data":"8ecb04f8fe991e94403dd37500469e55a1b6cc6fd342933c0196f7732fd06c9f"}
Sep 11 21:44:29.060153 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.060099    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj" event={"ID":"17a71664-0437-47fb-98a2-660db9026270","Type":"ContainerDied","Data":"f8a98ccc1c019d976fffc235ce7a393bca7df9e3624848bddcce78622f328bf7"}
Sep 11 21:44:29.060153 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.060120    2544 scope.go:117] "RemoveContainer" containerID="8ecb04f8fe991e94403dd37500469e55a1b6cc6fd342933c0196f7732fd06c9f"
Sep 11 21:44:29.060322 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.060309    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"
Sep 11 21:44:29.069551 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.069522    2544 scope.go:117] "RemoveContainer" containerID="11badb6589f2b145c96c6819ecfcfd8f8d687cea962f01e838c127421fff9d5e"
Sep 11 21:44:29.078769 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.078751    2544 scope.go:117] "RemoveContainer" containerID="8ecb04f8fe991e94403dd37500469e55a1b6cc6fd342933c0196f7732fd06c9f"
Sep 11 21:44:29.079038 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:44:29.079019    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ecb04f8fe991e94403dd37500469e55a1b6cc6fd342933c0196f7732fd06c9f\": container with ID starting with 8ecb04f8fe991e94403dd37500469e55a1b6cc6fd342933c0196f7732fd06c9f not found: ID does not exist" containerID="8ecb04f8fe991e94403dd37500469e55a1b6cc6fd342933c0196f7732fd06c9f"
Sep 11 21:44:29.079083 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.079046    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecb04f8fe991e94403dd37500469e55a1b6cc6fd342933c0196f7732fd06c9f"} err="failed to get container status \"8ecb04f8fe991e94403dd37500469e55a1b6cc6fd342933c0196f7732fd06c9f\": rpc error: code = NotFound desc = could not find container \"8ecb04f8fe991e94403dd37500469e55a1b6cc6fd342933c0196f7732fd06c9f\": container with ID starting with 8ecb04f8fe991e94403dd37500469e55a1b6cc6fd342933c0196f7732fd06c9f not found: ID does not exist"
Sep 11 21:44:29.079083 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.079061    2544 scope.go:117] "RemoveContainer" containerID="11badb6589f2b145c96c6819ecfcfd8f8d687cea962f01e838c127421fff9d5e"
Sep 11 21:44:29.079337 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:44:29.079313    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11badb6589f2b145c96c6819ecfcfd8f8d687cea962f01e838c127421fff9d5e\": container with ID starting with 11badb6589f2b145c96c6819ecfcfd8f8d687cea962f01e838c127421fff9d5e not found: ID does not exist" containerID="11badb6589f2b145c96c6819ecfcfd8f8d687cea962f01e838c127421fff9d5e"
Sep 11 21:44:29.079378 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.079344    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11badb6589f2b145c96c6819ecfcfd8f8d687cea962f01e838c127421fff9d5e"} err="failed to get container status \"11badb6589f2b145c96c6819ecfcfd8f8d687cea962f01e838c127421fff9d5e\": rpc error: code = NotFound desc = could not find container \"11badb6589f2b145c96c6819ecfcfd8f8d687cea962f01e838c127421fff9d5e\": container with ID starting with 11badb6589f2b145c96c6819ecfcfd8f8d687cea962f01e838c127421fff9d5e not found: ID does not exist"
Sep 11 21:44:29.083916 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.083893    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"]
Sep 11 21:44:29.087466 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.087444    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b2d1bd14-openshift-default-f56596db4-rflwj"]
Sep 11 21:44:29.097932 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.097909    2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a71664-0437-47fb-98a2-660db9026270-istio-podinfo\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:44:29.097932 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.097932    2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-credential-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:44:29.098038 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.097943    2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a71664-0437-47fb-98a2-660db9026270-istio-envoy\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:44:29.098038 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.097951    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sldp\" (UniqueName: \"kubernetes.io/projected/17a71664-0437-47fb-98a2-660db9026270-kube-api-access-4sldp\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:44:29.098038 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.097960    2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a71664-0437-47fb-98a2-660db9026270-istiod-ca-cert\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:44:29.098038 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.097968    2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a71664-0437-47fb-98a2-660db9026270-istio-token\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:44:29.592039 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:29.591999    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17a71664-0437-47fb-98a2-660db9026270" path="/var/lib/kubelet/pods/17a71664-0437-47fb-98a2-660db9026270/volumes"
Sep 11 21:44:33.022189 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:33.022158    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf"
Sep 11 21:44:33.083930 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:33.083900    2544 generic.go:356] "Generic (PLEG): container finished" podID="f597fc9d-2f8e-4047-970b-2d85ca4a928d" containerID="8622ed6d9b23ae897278a035a57464c8b84a341294ac4b19007db1af147a0607" exitCode=0
Sep 11 21:44:33.084101 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:33.083968    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9" event={"ID":"f597fc9d-2f8e-4047-970b-2d85ca4a928d","Type":"ContainerDied","Data":"8622ed6d9b23ae897278a035a57464c8b84a341294ac4b19007db1af147a0607"}
Sep 11 21:44:33.084496 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:33.084481    2544 scope.go:117] "RemoveContainer" containerID="8622ed6d9b23ae897278a035a57464c8b84a341294ac4b19007db1af147a0607"
Sep 11 21:44:33.085670 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:33.085645    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:33.085732 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:33.085713    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:33.085770 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:33.085745    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:34.090508 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:34.090471    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9" event={"ID":"f597fc9d-2f8e-4047-970b-2d85ca4a928d","Type":"ContainerStarted","Data":"efb2003c4258fd32043c6f0e12417f21f64d8eba06badd307b79e168075c3c14"}
Sep 11 21:44:34.797472 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:34.797439    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:34.797703 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:34.797680    2544 patch_prober.go:28] interesting pod/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 11 21:44:34.797763 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:34.797744    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9" podUID="f597fc9d-2f8e-4047-970b-2d85ca4a928d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 11 21:44:35.796679 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:35.796641    2544 patch_prober.go:28] interesting pod/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 11 21:44:35.797066 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:35.796701    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9" podUID="f597fc9d-2f8e-4047-970b-2d85ca4a928d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 11 21:44:36.796902 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:36.796873    2544 patch_prober.go:28] interesting pod/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 11 21:44:36.797275 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:36.796924    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9" podUID="f597fc9d-2f8e-4047-970b-2d85ca4a928d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 11 21:44:37.797283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:37.797179    2544 patch_prober.go:28] interesting pod/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 11 21:44:37.797283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:37.797254    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9" podUID="f597fc9d-2f8e-4047-970b-2d85ca4a928d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 11 21:44:38.796962 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:38.796931    2544 patch_prober.go:28] interesting pod/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 11 21:44:38.797186 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:38.797162    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9" podUID="f597fc9d-2f8e-4047-970b-2d85ca4a928d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 11 21:44:39.800703 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:39.800671    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:39.801129 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:39.800967    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:39.801557 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:39.801540    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:44:49.170350 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:49.170319    2544 generic.go:356] "Generic (PLEG): container finished" podID="e9bf8917-07c3-44a2-bd90-f39662474b48" containerID="22d10f024d5a1ca8ede79a5308c73e54593a001c0a25640972a87685f3082bf9" exitCode=0
Sep 11 21:44:49.170781 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:49.170370    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285" event={"ID":"e9bf8917-07c3-44a2-bd90-f39662474b48","Type":"ContainerDied","Data":"22d10f024d5a1ca8ede79a5308c73e54593a001c0a25640972a87685f3082bf9"}
Sep 11 21:44:49.170826 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:49.170785    2544 scope.go:117] "RemoveContainer" containerID="22d10f024d5a1ca8ede79a5308c73e54593a001c0a25640972a87685f3082bf9"
Sep 11 21:44:49.172040 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:49.172011    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:49.172112 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:49.172098    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:49.172150 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:49.172137    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:44:50.177108 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:50.177070    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285" event={"ID":"e9bf8917-07c3-44a2-bd90-f39662474b48","Type":"ContainerStarted","Data":"c8ad819ae508af3c0243e01d9a6c42326caecc162242ceeadcc3f62c3e4b07b0"}
Sep 11 21:44:50.267675 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:50.267632    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:50.267955 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:50.267934    2544 patch_prober.go:28] interesting pod/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 11 21:44:50.268011 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:50.267977    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285" podUID="e9bf8917-07c3-44a2-bd90-f39662474b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 11 21:44:51.268078 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:51.268040    2544 patch_prober.go:28] interesting pod/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 11 21:44:51.268499 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:51.268110    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285" podUID="e9bf8917-07c3-44a2-bd90-f39662474b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 11 21:44:51.469450 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:51.469414    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-762b41ed-h6g5h"]
Sep 11 21:44:51.469759 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:51.469748    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a71664-0437-47fb-98a2-660db9026270" containerName="istio-proxy"
Sep 11 21:44:51.469800 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:51.469761    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a71664-0437-47fb-98a2-660db9026270" containerName="istio-proxy"
Sep 11 21:44:51.469833 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:51.469827    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a71664-0437-47fb-98a2-660db9026270" containerName="istio-proxy"
Sep 11 21:44:51.469866 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:51.469837    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a71664-0437-47fb-98a2-660db9026270" containerName="istio-proxy"
Sep 11 21:44:51.473288 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:51.473267    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-762b41ed-h6g5h"
Sep 11 21:44:51.475487 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:51.475456    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:44:51.475606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:51.475514    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 21:44:51.480687 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:51.480658    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-762b41ed-h6g5h"]
Sep 11 21:44:51.635885 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:51.635773    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea740f0-0e21-4b2f-838e-de164f980fa2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-762b41ed-h6g5h\" (UID: \"eea740f0-0e21-4b2f-838e-de164f980fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-762b41ed-h6g5h"
Sep 11 21:44:51.738189 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:51.737300    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea740f0-0e21-4b2f-838e-de164f980fa2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-762b41ed-h6g5h\" (UID: \"eea740f0-0e21-4b2f-838e-de164f980fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-762b41ed-h6g5h"
Sep 11 21:44:51.738408 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:51.738130    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea740f0-0e21-4b2f-838e-de164f980fa2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-762b41ed-h6g5h\" (UID: \"eea740f0-0e21-4b2f-838e-de164f980fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-762b41ed-h6g5h"
Sep 11 21:44:51.786153 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:51.786114    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-762b41ed-h6g5h"
Sep 11 21:44:52.079998 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:52.079858    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-762b41ed-h6g5h"]
Sep 11 21:44:52.082053 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:44:52.082029    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea740f0_0e21_4b2f_838e_de164f980fa2.slice/crio-f8e7077667874ebff446e88060904041e7bb941a508bd1f9a917af0f8afab556 WatchSource:0}: Error finding container f8e7077667874ebff446e88060904041e7bb941a508bd1f9a917af0f8afab556: Status 404 returned error can't find the container with id f8e7077667874ebff446e88060904041e7bb941a508bd1f9a917af0f8afab556
Sep 11 21:44:52.188296 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:52.188260    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-762b41ed-h6g5h" event={"ID":"eea740f0-0e21-4b2f-838e-de164f980fa2","Type":"ContainerStarted","Data":"f8e7077667874ebff446e88060904041e7bb941a508bd1f9a917af0f8afab556"}
Sep 11 21:44:52.268081 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:52.268042    2544 patch_prober.go:28] interesting pod/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 11 21:44:52.268487 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:52.268095    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285" podUID="e9bf8917-07c3-44a2-bd90-f39662474b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 11 21:44:53.194474 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:53.194376    2544 generic.go:356] "Generic (PLEG): container finished" podID="eea740f0-0e21-4b2f-838e-de164f980fa2" containerID="7e31139980c1e7d7c85e73107ef79ef311b71af4eb2e9921ead843053a4166f5" exitCode=0
Sep 11 21:44:53.194626 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:53.194465    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-762b41ed-h6g5h" event={"ID":"eea740f0-0e21-4b2f-838e-de164f980fa2","Type":"ContainerDied","Data":"7e31139980c1e7d7c85e73107ef79ef311b71af4eb2e9921ead843053a4166f5"}
Sep 11 21:44:53.267401 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:53.267364    2544 patch_prober.go:28] interesting pod/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 11 21:44:53.267578 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:53.267425    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285" podUID="e9bf8917-07c3-44a2-bd90-f39662474b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 11 21:44:54.268111 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:54.268075    2544 patch_prober.go:28] interesting pod/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 11 21:44:54.268524 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:54.268148    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285" podUID="e9bf8917-07c3-44a2-bd90-f39662474b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 11 21:44:54.378202 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:54.378174    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-762b41ed-h6g5h"
Sep 11 21:44:54.470160 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:54.470052    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea740f0-0e21-4b2f-838e-de164f980fa2-maas-api-service-ca\") pod \"eea740f0-0e21-4b2f-838e-de164f980fa2\" (UID: \"eea740f0-0e21-4b2f-838e-de164f980fa2\") "
Sep 11 21:44:54.470465 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:54.470440    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea740f0-0e21-4b2f-838e-de164f980fa2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eea740f0-0e21-4b2f-838e-de164f980fa2" (UID: "eea740f0-0e21-4b2f-838e-de164f980fa2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:44:54.571996 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:54.571957    2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea740f0-0e21-4b2f-838e-de164f980fa2-maas-api-service-ca\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:44:55.205242 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:55.205190    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-762b41ed-h6g5h" event={"ID":"eea740f0-0e21-4b2f-838e-de164f980fa2","Type":"ContainerDied","Data":"f8e7077667874ebff446e88060904041e7bb941a508bd1f9a917af0f8afab556"}
Sep 11 21:44:55.205242 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:55.205214    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-762b41ed-h6g5h"
Sep 11 21:44:55.205459 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:55.205261    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8e7077667874ebff446e88060904041e7bb941a508bd1f9a917af0f8afab556"
Sep 11 21:44:55.267558 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:55.267519    2544 patch_prober.go:28] interesting pod/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 11 21:44:55.267712 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:55.267576    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285" podUID="e9bf8917-07c3-44a2-bd90-f39662474b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 11 21:44:56.042914 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:56.042872    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:56.273351 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:56.273321    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:56.274197 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:56.274177    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:44:58.650968 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:58.650924    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-762b41ed-h6g5h"]
Sep 11 21:44:58.655335 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:58.655306    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-762b41ed-h6g5h"]
Sep 11 21:44:58.791365 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:58.791332    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"]
Sep 11 21:44:58.791593 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:58.791562    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285" podUID="e9bf8917-07c3-44a2-bd90-f39662474b48" containerName="istio-proxy" containerID="cri-o://c8ad819ae508af3c0243e01d9a6c42326caecc162242ceeadcc3f62c3e4b07b0" gracePeriod=30
Sep 11 21:44:59.551392 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.551355    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"]
Sep 11 21:44:59.551754 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.551740    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a71664-0437-47fb-98a2-660db9026270" containerName="istio-proxy"
Sep 11 21:44:59.551754 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.551755    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a71664-0437-47fb-98a2-660db9026270" containerName="istio-proxy"
Sep 11 21:44:59.551844 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.551766    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea740f0-0e21-4b2f-838e-de164f980fa2" containerName="revoke-keys"
Sep 11 21:44:59.551844 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.551772    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea740f0-0e21-4b2f-838e-de164f980fa2" containerName="revoke-keys"
Sep 11 21:44:59.551903 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.551844    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="eea740f0-0e21-4b2f-838e-de164f980fa2" containerName="revoke-keys"
Sep 11 21:44:59.555424 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.555408    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.557867 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.557842    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-dad794b4-openshift-default-dockercfg-9w8f6\""
Sep 11 21:44:59.568075 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.568048    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"]
Sep 11 21:44:59.591965 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.591925    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eea740f0-0e21-4b2f-838e-de164f980fa2" path="/var/lib/kubelet/pods/eea740f0-0e21-4b2f-838e-de164f980fa2/volumes"
Sep 11 21:44:59.619052 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.619014    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-credential-socket\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.619230 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.619079    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1995f0-8282-4fee-b993-e9255506345e-istiod-ca-cert\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.619230 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.619125    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-istio-data\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.619230 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.619149    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf1995f0-8282-4fee-b993-e9255506345e-istio-token\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.619230 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.619176    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1995f0-8282-4fee-b993-e9255506345e-istio-podinfo\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.619370 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.619212    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwlhm\" (UniqueName: \"kubernetes.io/projected/bf1995f0-8282-4fee-b993-e9255506345e-kube-api-access-gwlhm\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.619370 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.619265    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-workload-certs\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.619370 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.619323    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-workload-socket\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.619370 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.619362    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-istio-envoy\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.720465 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.720425    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1995f0-8282-4fee-b993-e9255506345e-istiod-ca-cert\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.720977 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.720481    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-istio-data\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.720977 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.720508    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf1995f0-8282-4fee-b993-e9255506345e-istio-token\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.720977 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.720531    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1995f0-8282-4fee-b993-e9255506345e-istio-podinfo\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.720977 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.720558    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwlhm\" (UniqueName: \"kubernetes.io/projected/bf1995f0-8282-4fee-b993-e9255506345e-kube-api-access-gwlhm\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.720977 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.720582    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-workload-certs\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.720977 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.720621    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-workload-socket\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.720977 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.720649    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-istio-envoy\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.720977 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.720746    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-credential-socket\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.721402 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.721149    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-workload-certs\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.721402 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.721158    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-istio-data\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.721402 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.721307    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1995f0-8282-4fee-b993-e9255506345e-istiod-ca-cert\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.721529 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.721501    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-credential-socket\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.721582 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.721525    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-workload-socket\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.723317 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.723292    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-istio-envoy\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.723423 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.723293    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1995f0-8282-4fee-b993-e9255506345e-istio-podinfo\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.734553 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.734521    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf1995f0-8282-4fee-b993-e9255506345e-istio-token\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.738831 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.738805    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwlhm\" (UniqueName: \"kubernetes.io/projected/bf1995f0-8282-4fee-b993-e9255506345e-kube-api-access-gwlhm\") pod \"e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") " pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:44:59.867517 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:44:59.867415    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:45:00.139588 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.139498    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819385-vxjr6"]
Sep 11 21:45:00.145015 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.144989    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819385-vxjr6"
Sep 11 21:45:00.147796 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.147769    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-35e777-29819385-jn7qd"]
Sep 11 21:45:00.148939 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.148910    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 21:45:00.151906 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.151875    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7e6f5e-29819385-2qlm6"]
Sep 11 21:45:00.152033 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.152001    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-35e777-29819385-jn7qd"
Sep 11 21:45:00.160149 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.156650    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819385-vxjr6"]
Sep 11 21:45:00.160149 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.156764    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7e6f5e-29819385-2qlm6"
Sep 11 21:45:00.171864 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.170267    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819385-vxjr6"
Sep 11 21:45:00.174567 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.173579    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:45:00.174567 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.173664    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:45:00.174567 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.173704    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:45:00.174974 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.174949    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"]
Sep 11 21:45:00.178976 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.178461    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-35e777-29819385-jn7qd"]
Sep 11 21:45:00.181664 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.181380    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7e6f5e-29819385-2qlm6"]
Sep 11 21:45:00.184436 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.184415    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f12741-29819385-6zjsp"]
Sep 11 21:45:00.188841 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.188821    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f12741-29819385-6zjsp"
Sep 11 21:45:00.189747 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.189621    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f12741-29819385-6zjsp"]
Sep 11 21:45:00.190126 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.190111    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-35e777-29819385-jn7qd"
Sep 11 21:45:00.201334 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.201302    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7e6f5e-29819385-2qlm6"
Sep 11 21:45:00.216161 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.216116    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f12741-29819385-6zjsp"
Sep 11 21:45:00.234253 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.233681    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l" event={"ID":"bf1995f0-8282-4fee-b993-e9255506345e","Type":"ContainerStarted","Data":"df7ab3742103fdecfa90e83fe7f85eab6ab9621d6ca4a3e025638efb231da36e"}
Sep 11 21:45:00.464036 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.464006    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819385-vxjr6"]
Sep 11 21:45:00.465984 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:45:00.465955    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9211812_da2c_4a91_9c92_a1ede1639ad5.slice/crio-6199bc477718504315af0750183fd4e73a78b1fc1b6f30f00eac4225375448a6 WatchSource:0}: Error finding container 6199bc477718504315af0750183fd4e73a78b1fc1b6f30f00eac4225375448a6: Status 404 returned error can't find the container with id 6199bc477718504315af0750183fd4e73a78b1fc1b6f30f00eac4225375448a6
Sep 11 21:45:00.497150 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.497124    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-35e777-29819385-jn7qd"]
Sep 11 21:45:00.498629 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:45:00.498604    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0627f81_4b5c_49f5_8315_83a9630667a9.slice/crio-29e12b55226f8440f7b0a4ff427c3ee2eac97e94826780c0a60ad4a5ad558e13 WatchSource:0}: Error finding container 29e12b55226f8440f7b0a4ff427c3ee2eac97e94826780c0a60ad4a5ad558e13: Status 404 returned error can't find the container with id 29e12b55226f8440f7b0a4ff427c3ee2eac97e94826780c0a60ad4a5ad558e13
Sep 11 21:45:00.508403 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.508383    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7e6f5e-29819385-2qlm6"]
Sep 11 21:45:00.511580 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:45:00.511543    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98852096_4bda_4efc_9b23_ebea5edfcbc3.slice/crio-91effe1d3792a94ccdd640caba60d68aed8f55c6f2516bc6eff2ec62a01c047a WatchSource:0}: Error finding container 91effe1d3792a94ccdd640caba60d68aed8f55c6f2516bc6eff2ec62a01c047a: Status 404 returned error can't find the container with id 91effe1d3792a94ccdd640caba60d68aed8f55c6f2516bc6eff2ec62a01c047a
Sep 11 21:45:00.531967 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:00.531947    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f12741-29819385-6zjsp"]
Sep 11 21:45:00.534989 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:45:00.534616    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a025137_55cd_482f_82bf_63e9d4265b95.slice/crio-368eaa0369fc9b40bff6156cf6f62f8cdf3f835869f07c26cbe9616bec92ab41 WatchSource:0}: Error finding container 368eaa0369fc9b40bff6156cf6f62f8cdf3f835869f07c26cbe9616bec92ab41: Status 404 returned error can't find the container with id 368eaa0369fc9b40bff6156cf6f62f8cdf3f835869f07c26cbe9616bec92ab41
Sep 11 21:45:01.239853 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:01.239813    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f12741-29819385-6zjsp" event={"ID":"0a025137-55cd-482f-82bf-63e9d4265b95","Type":"ContainerStarted","Data":"6802469d941e81b22628fe519d27ad928325429b68da6c585044de4b5abab8e3"}
Sep 11 21:45:01.240320 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:01.239902    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f12741-29819385-6zjsp" event={"ID":"0a025137-55cd-482f-82bf-63e9d4265b95","Type":"ContainerStarted","Data":"368eaa0369fc9b40bff6156cf6f62f8cdf3f835869f07c26cbe9616bec92ab41"}
Sep 11 21:45:01.241417 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:01.241385    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-35e777-29819385-jn7qd" event={"ID":"c0627f81-4b5c-49f5-8315-83a9630667a9","Type":"ContainerStarted","Data":"d54f0b6ddab101409896991e48993633f85eeaa55eb2de66707c18e9bd9cc28d"}
Sep 11 21:45:01.241555 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:01.241425    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-35e777-29819385-jn7qd" event={"ID":"c0627f81-4b5c-49f5-8315-83a9630667a9","Type":"ContainerStarted","Data":"29e12b55226f8440f7b0a4ff427c3ee2eac97e94826780c0a60ad4a5ad558e13"}
Sep 11 21:45:01.242875 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:01.242845    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819385-vxjr6" event={"ID":"c9211812-da2c-4a91-9c92-a1ede1639ad5","Type":"ContainerStarted","Data":"5352e26355849edc3e50116ffaeae859a290f7715383187c1cdd15987ce1ef3d"}
Sep 11 21:45:01.242994 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:01.242890    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819385-vxjr6" event={"ID":"c9211812-da2c-4a91-9c92-a1ede1639ad5","Type":"ContainerStarted","Data":"6199bc477718504315af0750183fd4e73a78b1fc1b6f30f00eac4225375448a6"}
Sep 11 21:45:01.244368 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:01.244345    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l" event={"ID":"bf1995f0-8282-4fee-b993-e9255506345e","Type":"ContainerStarted","Data":"eb491b2185dec23d6c20b268f07f949960b019cb661c4812714e0a13d689a204"}
Sep 11 21:45:01.245729 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:01.245707    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7e6f5e-29819385-2qlm6" event={"ID":"98852096-4bda-4efc-9b23-ebea5edfcbc3","Type":"ContainerStarted","Data":"fe9d6d54d714c2cbd6c0e5d7fb4ce3ac781f5661ad9a404443bca31b1b50246c"}
Sep 11 21:45:01.245820 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:01.245748    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7e6f5e-29819385-2qlm6" event={"ID":"98852096-4bda-4efc-9b23-ebea5edfcbc3","Type":"ContainerStarted","Data":"91effe1d3792a94ccdd640caba60d68aed8f55c6f2516bc6eff2ec62a01c047a"}
Sep 11 21:45:01.253977 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:01.253913    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f12741-29819385-6zjsp" podStartSLOduration=1.253900494 podStartE2EDuration="1.253900494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:45:01.25303764 +0000 UTC m=+864.222454461" watchObservedRunningTime="2026-09-11 21:45:01.253900494 +0000 UTC m=+864.223317353"
Sep 11 21:45:01.267063 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:01.267023    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819385-vxjr6" podStartSLOduration=1.267008501 podStartE2EDuration="1.267008501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:45:01.26633112 +0000 UTC m=+864.235747940" watchObservedRunningTime="2026-09-11 21:45:01.267008501 +0000 UTC m=+864.236425320"
Sep 11 21:45:01.285818 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:01.285721    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l" podStartSLOduration=2.285706677 podStartE2EDuration="2.285706677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:45:01.283716093 +0000 UTC m=+864.253132914" watchObservedRunningTime="2026-09-11 21:45:01.285706677 +0000 UTC m=+864.255123497"
Sep 11 21:45:01.299430 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:01.299383    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7e6f5e-29819385-2qlm6" podStartSLOduration=1.2993713150000001 podStartE2EDuration="1.299371315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:45:01.297386048 +0000 UTC m=+864.266802870" watchObservedRunningTime="2026-09-11 21:45:01.299371315 +0000 UTC m=+864.268788159"
Sep 11 21:45:01.314387 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:01.314342    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-35e777-29819385-jn7qd" podStartSLOduration=1.314322908 podStartE2EDuration="1.314322908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:45:01.312268117 +0000 UTC m=+864.281684934" watchObservedRunningTime="2026-09-11 21:45:01.314322908 +0000 UTC m=+864.283739730"
Sep 11 21:45:01.868373 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:01.868333    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:45:01.873765 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:01.873736    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:45:02.253347 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:02.253313    2544 generic.go:356] "Generic (PLEG): container finished" podID="0a025137-55cd-482f-82bf-63e9d4265b95" containerID="6802469d941e81b22628fe519d27ad928325429b68da6c585044de4b5abab8e3" exitCode=0
Sep 11 21:45:02.253804 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:02.253385    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f12741-29819385-6zjsp" event={"ID":"0a025137-55cd-482f-82bf-63e9d4265b95","Type":"ContainerDied","Data":"6802469d941e81b22628fe519d27ad928325429b68da6c585044de4b5abab8e3"}
Sep 11 21:45:02.254903 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:02.254876    2544 generic.go:356] "Generic (PLEG): container finished" podID="c0627f81-4b5c-49f5-8315-83a9630667a9" containerID="d54f0b6ddab101409896991e48993633f85eeaa55eb2de66707c18e9bd9cc28d" exitCode=0
Sep 11 21:45:02.255006 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:02.254960    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-35e777-29819385-jn7qd" event={"ID":"c0627f81-4b5c-49f5-8315-83a9630667a9","Type":"ContainerDied","Data":"d54f0b6ddab101409896991e48993633f85eeaa55eb2de66707c18e9bd9cc28d"}
Sep 11 21:45:02.256584 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:02.256558    2544 generic.go:356] "Generic (PLEG): container finished" podID="c9211812-da2c-4a91-9c92-a1ede1639ad5" containerID="5352e26355849edc3e50116ffaeae859a290f7715383187c1cdd15987ce1ef3d" exitCode=0
Sep 11 21:45:02.256717 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:02.256659    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819385-vxjr6" event={"ID":"c9211812-da2c-4a91-9c92-a1ede1639ad5","Type":"ContainerDied","Data":"5352e26355849edc3e50116ffaeae859a290f7715383187c1cdd15987ce1ef3d"}
Sep 11 21:45:02.258129 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:02.258087    2544 generic.go:356] "Generic (PLEG): container finished" podID="98852096-4bda-4efc-9b23-ebea5edfcbc3" containerID="fe9d6d54d714c2cbd6c0e5d7fb4ce3ac781f5661ad9a404443bca31b1b50246c" exitCode=0
Sep 11 21:45:02.258250 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:02.258173    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7e6f5e-29819385-2qlm6" event={"ID":"98852096-4bda-4efc-9b23-ebea5edfcbc3","Type":"ContainerDied","Data":"fe9d6d54d714c2cbd6c0e5d7fb4ce3ac781f5661ad9a404443bca31b1b50246c"}
Sep 11 21:45:02.258442 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:02.258421    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:45:02.259375 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:02.259357    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:45:03.414952 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:03.414927    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7e6f5e-29819385-2qlm6"
Sep 11 21:45:03.481580 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:03.481552    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-35e777-29819385-jn7qd"
Sep 11 21:45:03.484968 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:03.484950    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819385-vxjr6"
Sep 11 21:45:03.513753 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:03.513675    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f12741-29819385-6zjsp"
Sep 11 21:45:04.038708 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.038678    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:45:04.061471 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.061437    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-podinfo\") pod \"e9bf8917-07c3-44a2-bd90-f39662474b48\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") "
Sep 11 21:45:04.061602 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.061507    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-data\") pod \"e9bf8917-07c3-44a2-bd90-f39662474b48\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") "
Sep 11 21:45:04.061643 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.061622    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-workload-socket\") pod \"e9bf8917-07c3-44a2-bd90-f39662474b48\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") "
Sep 11 21:45:04.061701 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.061666    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-envoy\") pod \"e9bf8917-07c3-44a2-bd90-f39662474b48\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") "
Sep 11 21:45:04.061754 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.061697    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-token\") pod \"e9bf8917-07c3-44a2-bd90-f39662474b48\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") "
Sep 11 21:45:04.061754 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.061737    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-workload-certs\") pod \"e9bf8917-07c3-44a2-bd90-f39662474b48\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") "
Sep 11 21:45:04.061856 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.061734    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-data" (OuterVolumeSpecName: "istio-data") pod "e9bf8917-07c3-44a2-bd90-f39662474b48" (UID: "e9bf8917-07c3-44a2-bd90-f39662474b48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:45:04.061856 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.061780    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-credential-socket\") pod \"e9bf8917-07c3-44a2-bd90-f39662474b48\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") "
Sep 11 21:45:04.061856 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.061850    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bf8917-07c3-44a2-bd90-f39662474b48-istiod-ca-cert\") pod \"e9bf8917-07c3-44a2-bd90-f39662474b48\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") "
Sep 11 21:45:04.061996 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.061899    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e9bf8917-07c3-44a2-bd90-f39662474b48" (UID: "e9bf8917-07c3-44a2-bd90-f39662474b48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:45:04.061996 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.061976    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e9bf8917-07c3-44a2-bd90-f39662474b48" (UID: "e9bf8917-07c3-44a2-bd90-f39662474b48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:45:04.062093 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.062075    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e9bf8917-07c3-44a2-bd90-f39662474b48" (UID: "e9bf8917-07c3-44a2-bd90-f39662474b48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:45:04.062215 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.062186    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btnl2\" (UniqueName: \"kubernetes.io/projected/e9bf8917-07c3-44a2-bd90-f39662474b48-kube-api-access-btnl2\") pod \"e9bf8917-07c3-44a2-bd90-f39662474b48\" (UID: \"e9bf8917-07c3-44a2-bd90-f39662474b48\") "
Sep 11 21:45:04.062366 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.062332    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9bf8917-07c3-44a2-bd90-f39662474b48-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e9bf8917-07c3-44a2-bd90-f39662474b48" (UID: "e9bf8917-07c3-44a2-bd90-f39662474b48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:45:04.062624 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.062602    2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-workload-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:04.062698 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.062629    2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-credential-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:04.062698 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.062645    2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bf8917-07c3-44a2-bd90-f39662474b48-istiod-ca-cert\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:04.062698 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.062659    2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-data\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:04.062698 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.062672    2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-workload-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:04.063797 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.063777    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e9bf8917-07c3-44a2-bd90-f39662474b48" (UID: "e9bf8917-07c3-44a2-bd90-f39662474b48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:45:04.063966 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.063825    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e9bf8917-07c3-44a2-bd90-f39662474b48" (UID: "e9bf8917-07c3-44a2-bd90-f39662474b48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:45:04.064166 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.064144    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-token" (OuterVolumeSpecName: "istio-token") pod "e9bf8917-07c3-44a2-bd90-f39662474b48" (UID: "e9bf8917-07c3-44a2-bd90-f39662474b48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:45:04.064530 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.064407    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9bf8917-07c3-44a2-bd90-f39662474b48-kube-api-access-btnl2" (OuterVolumeSpecName: "kube-api-access-btnl2") pod "e9bf8917-07c3-44a2-bd90-f39662474b48" (UID: "e9bf8917-07c3-44a2-bd90-f39662474b48"). InnerVolumeSpecName "kube-api-access-btnl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:45:04.164155 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.164066    2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-podinfo\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:04.164155 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.164098    2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-envoy\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:04.164155 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.164107    2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9bf8917-07c3-44a2-bd90-f39662474b48-istio-token\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:04.164155 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.164115    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btnl2\" (UniqueName: \"kubernetes.io/projected/e9bf8917-07c3-44a2-bd90-f39662474b48-kube-api-access-btnl2\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:04.270900 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.270862    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f12741-29819385-6zjsp" event={"ID":"0a025137-55cd-482f-82bf-63e9d4265b95","Type":"ContainerDied","Data":"368eaa0369fc9b40bff6156cf6f62f8cdf3f835869f07c26cbe9616bec92ab41"}
Sep 11 21:45:04.270900 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.270882    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f12741-29819385-6zjsp"
Sep 11 21:45:04.271119 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.270948    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="368eaa0369fc9b40bff6156cf6f62f8cdf3f835869f07c26cbe9616bec92ab41"
Sep 11 21:45:04.272279 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.272251    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-35e777-29819385-jn7qd" event={"ID":"c0627f81-4b5c-49f5-8315-83a9630667a9","Type":"ContainerDied","Data":"29e12b55226f8440f7b0a4ff427c3ee2eac97e94826780c0a60ad4a5ad558e13"}
Sep 11 21:45:04.272387 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.272302    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29e12b55226f8440f7b0a4ff427c3ee2eac97e94826780c0a60ad4a5ad558e13"
Sep 11 21:45:04.272387 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.272261    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-35e777-29819385-jn7qd"
Sep 11 21:45:04.273817 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.273798    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819385-vxjr6"
Sep 11 21:45:04.273817 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.273807    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819385-vxjr6" event={"ID":"c9211812-da2c-4a91-9c92-a1ede1639ad5","Type":"ContainerDied","Data":"6199bc477718504315af0750183fd4e73a78b1fc1b6f30f00eac4225375448a6"}
Sep 11 21:45:04.273994 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.273844    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6199bc477718504315af0750183fd4e73a78b1fc1b6f30f00eac4225375448a6"
Sep 11 21:45:04.275750 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.275721    2544 generic.go:356] "Generic (PLEG): container finished" podID="e9bf8917-07c3-44a2-bd90-f39662474b48" containerID="c8ad819ae508af3c0243e01d9a6c42326caecc162242ceeadcc3f62c3e4b07b0" exitCode=0
Sep 11 21:45:04.275892 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.275870    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285" event={"ID":"e9bf8917-07c3-44a2-bd90-f39662474b48","Type":"ContainerDied","Data":"c8ad819ae508af3c0243e01d9a6c42326caecc162242ceeadcc3f62c3e4b07b0"}
Sep 11 21:45:04.275957 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.275915    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285" event={"ID":"e9bf8917-07c3-44a2-bd90-f39662474b48","Type":"ContainerDied","Data":"42e31542a0d05fdc5c76dd1128cfdbed4c33115db9d0b4b7b8069b5093adc4ad"}
Sep 11 21:45:04.275957 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.275947    2544 scope.go:117] "RemoveContainer" containerID="c8ad819ae508af3c0243e01d9a6c42326caecc162242ceeadcc3f62c3e4b07b0"
Sep 11 21:45:04.276440 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.275942    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"
Sep 11 21:45:04.277444 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.277427    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7e6f5e-29819385-2qlm6"
Sep 11 21:45:04.277528 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.277508    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7e6f5e-29819385-2qlm6" event={"ID":"98852096-4bda-4efc-9b23-ebea5edfcbc3","Type":"ContainerDied","Data":"91effe1d3792a94ccdd640caba60d68aed8f55c6f2516bc6eff2ec62a01c047a"}
Sep 11 21:45:04.277595 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.277546    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91effe1d3792a94ccdd640caba60d68aed8f55c6f2516bc6eff2ec62a01c047a"
Sep 11 21:45:04.284989 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.284972    2544 scope.go:117] "RemoveContainer" containerID="22d10f024d5a1ca8ede79a5308c73e54593a001c0a25640972a87685f3082bf9"
Sep 11 21:45:04.292784 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.292766    2544 scope.go:117] "RemoveContainer" containerID="c8ad819ae508af3c0243e01d9a6c42326caecc162242ceeadcc3f62c3e4b07b0"
Sep 11 21:45:04.293061 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:45:04.293041    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8ad819ae508af3c0243e01d9a6c42326caecc162242ceeadcc3f62c3e4b07b0\": container with ID starting with c8ad819ae508af3c0243e01d9a6c42326caecc162242ceeadcc3f62c3e4b07b0 not found: ID does not exist" containerID="c8ad819ae508af3c0243e01d9a6c42326caecc162242ceeadcc3f62c3e4b07b0"
Sep 11 21:45:04.293110 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.293069    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ad819ae508af3c0243e01d9a6c42326caecc162242ceeadcc3f62c3e4b07b0"} err="failed to get container status \"c8ad819ae508af3c0243e01d9a6c42326caecc162242ceeadcc3f62c3e4b07b0\": rpc error: code = NotFound desc = could not find container \"c8ad819ae508af3c0243e01d9a6c42326caecc162242ceeadcc3f62c3e4b07b0\": container with ID starting with c8ad819ae508af3c0243e01d9a6c42326caecc162242ceeadcc3f62c3e4b07b0 not found: ID does not exist"
Sep 11 21:45:04.293110 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.293084    2544 scope.go:117] "RemoveContainer" containerID="22d10f024d5a1ca8ede79a5308c73e54593a001c0a25640972a87685f3082bf9"
Sep 11 21:45:04.293383 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:45:04.293362    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d10f024d5a1ca8ede79a5308c73e54593a001c0a25640972a87685f3082bf9\": container with ID starting with 22d10f024d5a1ca8ede79a5308c73e54593a001c0a25640972a87685f3082bf9 not found: ID does not exist" containerID="22d10f024d5a1ca8ede79a5308c73e54593a001c0a25640972a87685f3082bf9"
Sep 11 21:45:04.293437 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.293389    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d10f024d5a1ca8ede79a5308c73e54593a001c0a25640972a87685f3082bf9"} err="failed to get container status \"22d10f024d5a1ca8ede79a5308c73e54593a001c0a25640972a87685f3082bf9\": rpc error: code = NotFound desc = could not find container \"22d10f024d5a1ca8ede79a5308c73e54593a001c0a25640972a87685f3082bf9\": container with ID starting with 22d10f024d5a1ca8ede79a5308c73e54593a001c0a25640972a87685f3082bf9 not found: ID does not exist"
Sep 11 21:45:04.347343 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.347309    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"]
Sep 11 21:45:04.350698 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.350670    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-762b41ed-gw-openshift-default-64c6994548-qk285"]
Sep 11 21:45:04.718976 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.718940    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d"]
Sep 11 21:45:04.719404 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.719294    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9211812-da2c-4a91-9c92-a1ede1639ad5" containerName="cleanup"
Sep 11 21:45:04.719404 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.719306    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9211812-da2c-4a91-9c92-a1ede1639ad5" containerName="cleanup"
Sep 11 21:45:04.719404 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.719315    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a025137-55cd-482f-82bf-63e9d4265b95" containerName="cleanup"
Sep 11 21:45:04.719404 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.719320    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a025137-55cd-482f-82bf-63e9d4265b95" containerName="cleanup"
Sep 11 21:45:04.719404 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.719332    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0627f81-4b5c-49f5-8315-83a9630667a9" containerName="cleanup"
Sep 11 21:45:04.719404 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.719338    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0627f81-4b5c-49f5-8315-83a9630667a9" containerName="cleanup"
Sep 11 21:45:04.719404 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.719348    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9bf8917-07c3-44a2-bd90-f39662474b48" containerName="istio-proxy"
Sep 11 21:45:04.719404 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.719355    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9bf8917-07c3-44a2-bd90-f39662474b48" containerName="istio-proxy"
Sep 11 21:45:04.719404 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.719365    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9bf8917-07c3-44a2-bd90-f39662474b48" containerName="istio-proxy"
Sep 11 21:45:04.719404 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.719371    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9bf8917-07c3-44a2-bd90-f39662474b48" containerName="istio-proxy"
Sep 11 21:45:04.719404 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.719380    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98852096-4bda-4efc-9b23-ebea5edfcbc3" containerName="cleanup"
Sep 11 21:45:04.719404 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.719385    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="98852096-4bda-4efc-9b23-ebea5edfcbc3" containerName="cleanup"
Sep 11 21:45:04.719860 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.719439    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="98852096-4bda-4efc-9b23-ebea5edfcbc3" containerName="cleanup"
Sep 11 21:45:04.719860 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.719448    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0627f81-4b5c-49f5-8315-83a9630667a9" containerName="cleanup"
Sep 11 21:45:04.719860 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.719454    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a025137-55cd-482f-82bf-63e9d4265b95" containerName="cleanup"
Sep 11 21:45:04.719860 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.719459    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9bf8917-07c3-44a2-bd90-f39662474b48" containerName="istio-proxy"
Sep 11 21:45:04.719860 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.719465    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9211812-da2c-4a91-9c92-a1ede1639ad5" containerName="cleanup"
Sep 11 21:45:04.719860 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.719471    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9bf8917-07c3-44a2-bd90-f39662474b48" containerName="istio-proxy"
Sep 11 21:45:04.723912 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.723893    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d"
Sep 11 21:45:04.726389 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.726358    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-dad794b4\""
Sep 11 21:45:04.726389 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.726379    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-dad794b4-dockercfg-wf9vn\""
Sep 11 21:45:04.735598 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.735576    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d"]
Sep 11 21:45:04.746906 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.746878    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr"]
Sep 11 21:45:04.750192 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.750175    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr"
Sep 11 21:45:04.761032 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.761008    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr"]
Sep 11 21:45:04.770545 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.770518    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3629dec8-2b67-460b-96d7-21e5d2e4f6ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d\" (UID: \"3629dec8-2b67-460b-96d7-21e5d2e4f6ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d"
Sep 11 21:45:04.770660 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.770554    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46bh9\" (UniqueName: \"kubernetes.io/projected/3629dec8-2b67-460b-96d7-21e5d2e4f6ea-kube-api-access-46bh9\") pod \"payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d\" (UID: \"3629dec8-2b67-460b-96d7-21e5d2e4f6ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d"
Sep 11 21:45:04.770660 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.770633    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4wn6\" (UniqueName: \"kubernetes.io/projected/ade1c035-2822-4b9a-9222-84c59a32c668-kube-api-access-q4wn6\") pod \"payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr\" (UID: \"ade1c035-2822-4b9a-9222-84c59a32c668\") " pod="openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr"
Sep 11 21:45:04.770748 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.770660    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade1c035-2822-4b9a-9222-84c59a32c668-plugins-config-volume\") pod \"payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr\" (UID: \"ade1c035-2822-4b9a-9222-84c59a32c668\") " pod="openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr"
Sep 11 21:45:04.871608 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.871574    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46bh9\" (UniqueName: \"kubernetes.io/projected/3629dec8-2b67-460b-96d7-21e5d2e4f6ea-kube-api-access-46bh9\") pod \"payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d\" (UID: \"3629dec8-2b67-460b-96d7-21e5d2e4f6ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d"
Sep 11 21:45:04.871761 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.871675    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4wn6\" (UniqueName: \"kubernetes.io/projected/ade1c035-2822-4b9a-9222-84c59a32c668-kube-api-access-q4wn6\") pod \"payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr\" (UID: \"ade1c035-2822-4b9a-9222-84c59a32c668\") " pod="openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr"
Sep 11 21:45:04.871761 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.871715    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade1c035-2822-4b9a-9222-84c59a32c668-plugins-config-volume\") pod \"payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr\" (UID: \"ade1c035-2822-4b9a-9222-84c59a32c668\") " pod="openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr"
Sep 11 21:45:04.871833 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.871803    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3629dec8-2b67-460b-96d7-21e5d2e4f6ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d\" (UID: \"3629dec8-2b67-460b-96d7-21e5d2e4f6ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d"
Sep 11 21:45:04.872483 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.872464    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade1c035-2822-4b9a-9222-84c59a32c668-plugins-config-volume\") pod \"payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr\" (UID: \"ade1c035-2822-4b9a-9222-84c59a32c668\") " pod="openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr"
Sep 11 21:45:04.872524 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.872509    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3629dec8-2b67-460b-96d7-21e5d2e4f6ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d\" (UID: \"3629dec8-2b67-460b-96d7-21e5d2e4f6ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d"
Sep 11 21:45:04.880948 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.880918    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4wn6\" (UniqueName: \"kubernetes.io/projected/ade1c035-2822-4b9a-9222-84c59a32c668-kube-api-access-q4wn6\") pod \"payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr\" (UID: \"ade1c035-2822-4b9a-9222-84c59a32c668\") " pod="openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr"
Sep 11 21:45:04.881059 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:04.880925    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46bh9\" (UniqueName: \"kubernetes.io/projected/3629dec8-2b67-460b-96d7-21e5d2e4f6ea-kube-api-access-46bh9\") pod \"payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d\" (UID: \"3629dec8-2b67-460b-96d7-21e5d2e4f6ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d"
Sep 11 21:45:05.033959 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:05.033909    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d"
Sep 11 21:45:05.060445 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:05.060407    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr"
Sep 11 21:45:05.310072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:05.310040    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d"]
Sep 11 21:45:05.311353 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:45:05.311323    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3629dec8_2b67_460b_96d7_21e5d2e4f6ea.slice/crio-700b969fe26a55502ab3d1d20410fe6b58e3e1a3361ba2c06807ec731aa9ebde WatchSource:0}: Error finding container 700b969fe26a55502ab3d1d20410fe6b58e3e1a3361ba2c06807ec731aa9ebde: Status 404 returned error can't find the container with id 700b969fe26a55502ab3d1d20410fe6b58e3e1a3361ba2c06807ec731aa9ebde
Sep 11 21:45:05.341145 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:05.341117    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr"]
Sep 11 21:45:05.342508 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:45:05.342481    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade1c035_2822_4b9a_9222_84c59a32c668.slice/crio-be543ea2d67b59b1c9c63eaa0a9ef7f8930a2f27feb2986a10ad65f666fd2759 WatchSource:0}: Error finding container be543ea2d67b59b1c9c63eaa0a9ef7f8930a2f27feb2986a10ad65f666fd2759: Status 404 returned error can't find the container with id be543ea2d67b59b1c9c63eaa0a9ef7f8930a2f27feb2986a10ad65f666fd2759
Sep 11 21:45:05.593360 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:05.593262    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9bf8917-07c3-44a2-bd90-f39662474b48" path="/var/lib/kubelet/pods/e9bf8917-07c3-44a2-bd90-f39662474b48/volumes"
Sep 11 21:45:06.291832 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:06.291790    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d" event={"ID":"3629dec8-2b67-460b-96d7-21e5d2e4f6ea","Type":"ContainerStarted","Data":"98e4d68cf1b3508362662748c6e38d845dac31f8217373d9d223d312ae256724"}
Sep 11 21:45:06.292407 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:06.291972    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d"
Sep 11 21:45:06.292407 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:06.292000    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d" event={"ID":"3629dec8-2b67-460b-96d7-21e5d2e4f6ea","Type":"ContainerStarted","Data":"700b969fe26a55502ab3d1d20410fe6b58e3e1a3361ba2c06807ec731aa9ebde"}
Sep 11 21:45:06.293364 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:06.293341    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr" event={"ID":"ade1c035-2822-4b9a-9222-84c59a32c668","Type":"ContainerStarted","Data":"2449a85a68b651a2b303903cdf62d4a2391148c85c19d6339f73055dd5b40f57"}
Sep 11 21:45:06.293440 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:06.293382    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr" event={"ID":"ade1c035-2822-4b9a-9222-84c59a32c668","Type":"ContainerStarted","Data":"be543ea2d67b59b1c9c63eaa0a9ef7f8930a2f27feb2986a10ad65f666fd2759"}
Sep 11 21:45:06.293440 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:06.293402    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr"
Sep 11 21:45:06.308741 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:06.308701    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d" podStartSLOduration=2.308688597 podStartE2EDuration="2.308688597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:45:06.306586069 +0000 UTC m=+869.276002888" watchObservedRunningTime="2026-09-11 21:45:06.308688597 +0000 UTC m=+869.278105413"
Sep 11 21:45:06.322782 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:06.322730    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr" podStartSLOduration=2.32271263 podStartE2EDuration="2.32271263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:45:06.322434172 +0000 UTC m=+869.291850991" watchObservedRunningTime="2026-09-11 21:45:06.32271263 +0000 UTC m=+869.292129451"
Sep 11 21:45:07.246424 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.246388    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"]
Sep 11 21:45:07.250169 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.250151    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.253038 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.253013    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-35e777\"/\"kube-root-ca.crt\""
Sep 11 21:45:07.254053 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.254032    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-35e777\"/\"default-dockercfg-ctc57\""
Sep 11 21:45:07.254185 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.254078    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-35e777\"/\"openshift-service-ca.crt\""
Sep 11 21:45:07.254185 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.254090    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-35e777\"/\"simulator-w4-35e777-kserve-self-signed-certs\""
Sep 11 21:45:07.261677 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.261655    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"]
Sep 11 21:45:07.295761 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.295730    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-home\") pod \"simulator-w4-35e777-kserve-56b754cc95-r75vs\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") " pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.296122 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.295768    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508b6f76-c334-4096-9d26-1c4000534efe-tls-certs\") pod \"simulator-w4-35e777-kserve-56b754cc95-r75vs\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") " pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.296122 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.295800    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5qmj\" (UniqueName: \"kubernetes.io/projected/508b6f76-c334-4096-9d26-1c4000534efe-kube-api-access-l5qmj\") pod \"simulator-w4-35e777-kserve-56b754cc95-r75vs\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") " pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.296122 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.295869    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-model-cache\") pod \"simulator-w4-35e777-kserve-56b754cc95-r75vs\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") " pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.296122 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.295967    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-dshm\") pod \"simulator-w4-35e777-kserve-56b754cc95-r75vs\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") " pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.296122 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.296043    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-tmp-dir\") pod \"simulator-w4-35e777-kserve-56b754cc95-r75vs\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") " pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.397504 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.397461    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5qmj\" (UniqueName: \"kubernetes.io/projected/508b6f76-c334-4096-9d26-1c4000534efe-kube-api-access-l5qmj\") pod \"simulator-w4-35e777-kserve-56b754cc95-r75vs\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") " pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.397683 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.397512    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-model-cache\") pod \"simulator-w4-35e777-kserve-56b754cc95-r75vs\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") " pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.397683 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.397659    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-dshm\") pod \"simulator-w4-35e777-kserve-56b754cc95-r75vs\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") " pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.397862 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.397833    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-tmp-dir\") pod \"simulator-w4-35e777-kserve-56b754cc95-r75vs\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") " pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.397949 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.397903    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-model-cache\") pod \"simulator-w4-35e777-kserve-56b754cc95-r75vs\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") " pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.398013 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.397985    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-home\") pod \"simulator-w4-35e777-kserve-56b754cc95-r75vs\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") " pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.398013 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.398006    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508b6f76-c334-4096-9d26-1c4000534efe-tls-certs\") pod \"simulator-w4-35e777-kserve-56b754cc95-r75vs\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") " pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.398144 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.398118    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-tmp-dir\") pod \"simulator-w4-35e777-kserve-56b754cc95-r75vs\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") " pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.398363 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.398333    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-home\") pod \"simulator-w4-35e777-kserve-56b754cc95-r75vs\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") " pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.400313 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.400289    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-dshm\") pod \"simulator-w4-35e777-kserve-56b754cc95-r75vs\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") " pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.400531 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.400511    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508b6f76-c334-4096-9d26-1c4000534efe-tls-certs\") pod \"simulator-w4-35e777-kserve-56b754cc95-r75vs\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") " pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.405827 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.405807    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5qmj\" (UniqueName: \"kubernetes.io/projected/508b6f76-c334-4096-9d26-1c4000534efe-kube-api-access-l5qmj\") pod \"simulator-w4-35e777-kserve-56b754cc95-r75vs\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") " pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.561330 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.561209    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:07.832399 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:07.832370    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"]
Sep 11 21:45:07.834271 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:45:07.834240    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod508b6f76_c334_4096_9d26_1c4000534efe.slice/crio-11a93a8e7f9b1264be7e778577b08263497dfd7e78dc7b2adef3f15a697363e0 WatchSource:0}: Error finding container 11a93a8e7f9b1264be7e778577b08263497dfd7e78dc7b2adef3f15a697363e0: Status 404 returned error can't find the container with id 11a93a8e7f9b1264be7e778577b08263497dfd7e78dc7b2adef3f15a697363e0
Sep 11 21:45:08.306485 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:08.306451    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs" event={"ID":"508b6f76-c334-4096-9d26-1c4000534efe","Type":"ContainerStarted","Data":"5e530a8b0e9b7ff2c586ff6b36960de98928e8d8e75d89ee22252943fee82d3e"}
Sep 11 21:45:08.306485 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:08.306491    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs" event={"ID":"508b6f76-c334-4096-9d26-1c4000534efe","Type":"ContainerStarted","Data":"11a93a8e7f9b1264be7e778577b08263497dfd7e78dc7b2adef3f15a697363e0"}
Sep 11 21:45:08.325379 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:08.325324    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs" podStartSLOduration=1.325303557 podStartE2EDuration="1.325303557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:45:08.324064526 +0000 UTC m=+871.293481348" watchObservedRunningTime="2026-09-11 21:45:08.325303557 +0000 UTC m=+871.294720376"
Sep 11 21:45:17.302388 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:17.302356    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d"
Sep 11 21:45:17.302767 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:17.302427    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr"
Sep 11 21:45:17.561513 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:17.561427    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:17.561513 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:17.561480    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:17.574189 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:17.574162    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:18.375246 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:18.375197    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:45:23.393897 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:23.393810    2544 generic.go:356] "Generic (PLEG): container finished" podID="bf1995f0-8282-4fee-b993-e9255506345e" containerID="eb491b2185dec23d6c20b268f07f949960b019cb661c4812714e0a13d689a204" exitCode=0
Sep 11 21:45:23.393897 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:23.393884    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l" event={"ID":"bf1995f0-8282-4fee-b993-e9255506345e","Type":"ContainerDied","Data":"eb491b2185dec23d6c20b268f07f949960b019cb661c4812714e0a13d689a204"}
Sep 11 21:45:23.394352 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:23.394325    2544 scope.go:117] "RemoveContainer" containerID="eb491b2185dec23d6c20b268f07f949960b019cb661c4812714e0a13d689a204"
Sep 11 21:45:23.395500 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:23.395467    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:45:23.395610 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:23.395555    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:45:23.395678 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:23.395607    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:45:24.401021 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:24.400984    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l" event={"ID":"bf1995f0-8282-4fee-b993-e9255506345e","Type":"ContainerStarted","Data":"fef65d5b933945664f32bbeed585106feb9ece82709eab0659697b5a0711557b"}
Sep 11 21:45:24.868243 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:24.868169    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:45:24.868438 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:24.868415    2544 patch_prober.go:28] interesting pod/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 11 21:45:24.868492 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:24.868464    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l" podUID="bf1995f0-8282-4fee-b993-e9255506345e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 11 21:45:25.253893 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.253860    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"]
Sep 11 21:45:25.256353 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.256335    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.259829 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.259805    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-5b160c\"/\"default-dockercfg-wvks9\""
Sep 11 21:45:25.259970 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.259854    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-5b160c\"/\"autores-model-5b160c-kserve-self-signed-certs\""
Sep 11 21:45:25.259970 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.259912    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-5b160c\"/\"openshift-service-ca.crt\""
Sep 11 21:45:25.259970 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.259930    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-5b160c\"/\"kube-root-ca.crt\""
Sep 11 21:45:25.266335 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.266312    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"]
Sep 11 21:45:25.405355 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.405322    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-home\") pod \"autores-model-5b160c-kserve-5bfdc9dd75-xcc8z\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") " pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.405355 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.405366    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-tmp-dir\") pod \"autores-model-5b160c-kserve-5bfdc9dd75-xcc8z\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") " pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.405783 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.405404    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-model-cache\") pod \"autores-model-5b160c-kserve-5bfdc9dd75-xcc8z\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") " pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.405783 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.405448    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-dshm\") pod \"autores-model-5b160c-kserve-5bfdc9dd75-xcc8z\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") " pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.405783 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.405465    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpq8k\" (UniqueName: \"kubernetes.io/projected/0e9a0319-3733-43f4-9a57-f8497035fd1a-kube-api-access-wpq8k\") pod \"autores-model-5b160c-kserve-5bfdc9dd75-xcc8z\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") " pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.405783 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.405574    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e9a0319-3733-43f4-9a57-f8497035fd1a-tls-certs\") pod \"autores-model-5b160c-kserve-5bfdc9dd75-xcc8z\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") " pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.506336 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.506207    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e9a0319-3733-43f4-9a57-f8497035fd1a-tls-certs\") pod \"autores-model-5b160c-kserve-5bfdc9dd75-xcc8z\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") " pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.506336 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.506319    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-home\") pod \"autores-model-5b160c-kserve-5bfdc9dd75-xcc8z\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") " pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.506588 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.506354    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-tmp-dir\") pod \"autores-model-5b160c-kserve-5bfdc9dd75-xcc8z\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") " pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.506588 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.506400    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-model-cache\") pod \"autores-model-5b160c-kserve-5bfdc9dd75-xcc8z\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") " pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.506588 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.506435    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-dshm\") pod \"autores-model-5b160c-kserve-5bfdc9dd75-xcc8z\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") " pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.506588 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.506463    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpq8k\" (UniqueName: \"kubernetes.io/projected/0e9a0319-3733-43f4-9a57-f8497035fd1a-kube-api-access-wpq8k\") pod \"autores-model-5b160c-kserve-5bfdc9dd75-xcc8z\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") " pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.506831 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.506806    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-home\") pod \"autores-model-5b160c-kserve-5bfdc9dd75-xcc8z\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") " pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.506890 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.506850    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-model-cache\") pod \"autores-model-5b160c-kserve-5bfdc9dd75-xcc8z\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") " pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.507086 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.507067    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-tmp-dir\") pod \"autores-model-5b160c-kserve-5bfdc9dd75-xcc8z\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") " pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.508713 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.508684    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-dshm\") pod \"autores-model-5b160c-kserve-5bfdc9dd75-xcc8z\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") " pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.508813 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.508788    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e9a0319-3733-43f4-9a57-f8497035fd1a-tls-certs\") pod \"autores-model-5b160c-kserve-5bfdc9dd75-xcc8z\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") " pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.516601 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.516578    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpq8k\" (UniqueName: \"kubernetes.io/projected/0e9a0319-3733-43f4-9a57-f8497035fd1a-kube-api-access-wpq8k\") pod \"autores-model-5b160c-kserve-5bfdc9dd75-xcc8z\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") " pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.566488 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.566439    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:25.864899 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.864874    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"]
Sep 11 21:45:25.866793 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:45:25.866763    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e9a0319_3733_43f4_9a57_f8497035fd1a.slice/crio-b3a827a8d10850cdd86a912c7281d5c32cb9064caac53a720a6b950b3ada4efd WatchSource:0}: Error finding container b3a827a8d10850cdd86a912c7281d5c32cb9064caac53a720a6b950b3ada4efd: Status 404 returned error can't find the container with id b3a827a8d10850cdd86a912c7281d5c32cb9064caac53a720a6b950b3ada4efd
Sep 11 21:45:25.868103 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.868078    2544 patch_prober.go:28] interesting pod/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 11 21:45:25.868228 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:25.868139    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l" podUID="bf1995f0-8282-4fee-b993-e9255506345e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 11 21:45:26.413128 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:26.413089    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z" event={"ID":"0e9a0319-3733-43f4-9a57-f8497035fd1a","Type":"ContainerStarted","Data":"52f2449c2c1770eea56b0a6aa600ce550a5291dfa55b3ec343345fe01bc2e9ef"}
Sep 11 21:45:26.413128 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:26.413138    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z" event={"ID":"0e9a0319-3733-43f4-9a57-f8497035fd1a","Type":"ContainerStarted","Data":"b3a827a8d10850cdd86a912c7281d5c32cb9064caac53a720a6b950b3ada4efd"}
Sep 11 21:45:26.432671 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:26.432600    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z" podStartSLOduration=1.432576689 podStartE2EDuration="1.432576689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:45:26.429682519 +0000 UTC m=+889.399099354" watchObservedRunningTime="2026-09-11 21:45:26.432576689 +0000 UTC m=+889.401993512"
Sep 11 21:45:26.868795 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:26.868755    2544 patch_prober.go:28] interesting pod/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 11 21:45:26.869000 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:26.868818    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l" podUID="bf1995f0-8282-4fee-b993-e9255506345e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 11 21:45:27.868206 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:27.868155    2544 patch_prober.go:28] interesting pod/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 11 21:45:27.868642 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:27.868260    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l" podUID="bf1995f0-8282-4fee-b993-e9255506345e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 11 21:45:28.422355 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:28.422324    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dad794b4-7rtj8"]
Sep 11 21:45:28.425604 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:28.425576    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dad794b4-7rtj8"
Sep 11 21:45:28.430581 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:28.430550    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 21:45:28.430942 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:28.430669    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:45:28.435909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:28.435884    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dad794b4-7rtj8"]
Sep 11 21:45:28.443136 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:28.443098    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24515852-1d43-491f-a428-3282e7bac38d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dad794b4-7rtj8\" (UID: \"24515852-1d43-491f-a428-3282e7bac38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dad794b4-7rtj8"
Sep 11 21:45:28.544201 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:28.544157    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24515852-1d43-491f-a428-3282e7bac38d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dad794b4-7rtj8\" (UID: \"24515852-1d43-491f-a428-3282e7bac38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dad794b4-7rtj8"
Sep 11 21:45:28.545073 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:28.545035    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24515852-1d43-491f-a428-3282e7bac38d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dad794b4-7rtj8\" (UID: \"24515852-1d43-491f-a428-3282e7bac38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dad794b4-7rtj8"
Sep 11 21:45:28.742249 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:28.742196    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dad794b4-7rtj8"
Sep 11 21:45:28.868282 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:28.868236    2544 patch_prober.go:28] interesting pod/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 11 21:45:28.868647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:28.868313    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l" podUID="bf1995f0-8282-4fee-b993-e9255506345e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 11 21:45:29.013503 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:29.013475    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dad794b4-7rtj8"]
Sep 11 21:45:29.014965 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:45:29.014936    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24515852_1d43_491f_a428_3282e7bac38d.slice/crio-636b8f3edf89a50d1fbd0947bc8b0d2ec48cbf11f4a1d6ad4784b14aa6d36467 WatchSource:0}: Error finding container 636b8f3edf89a50d1fbd0947bc8b0d2ec48cbf11f4a1d6ad4784b14aa6d36467: Status 404 returned error can't find the container with id 636b8f3edf89a50d1fbd0947bc8b0d2ec48cbf11f4a1d6ad4784b14aa6d36467
Sep 11 21:45:29.446250 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:29.445309    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dad794b4-7rtj8" event={"ID":"24515852-1d43-491f-a428-3282e7bac38d","Type":"ContainerStarted","Data":"636b8f3edf89a50d1fbd0947bc8b0d2ec48cbf11f4a1d6ad4784b14aa6d36467"}
Sep 11 21:45:29.868009 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:29.867964    2544 patch_prober.go:28] interesting pod/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 11 21:45:29.868205 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:29.868027    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l" podUID="bf1995f0-8282-4fee-b993-e9255506345e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 11 21:45:30.453289 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:30.453196    2544 generic.go:356] "Generic (PLEG): container finished" podID="24515852-1d43-491f-a428-3282e7bac38d" containerID="ed374633dc24a7ce86e8b65f5ee2fcb40bb47d2b6fc1141ae3418312773637b6" exitCode=0
Sep 11 21:45:30.453289 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:30.453274    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dad794b4-7rtj8" event={"ID":"24515852-1d43-491f-a428-3282e7bac38d","Type":"ContainerDied","Data":"ed374633dc24a7ce86e8b65f5ee2fcb40bb47d2b6fc1141ae3418312773637b6"}
Sep 11 21:45:30.872050 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:30.872018    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:45:30.872318 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:30.872298    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:45:30.873022 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:30.873004    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:45:31.598253 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:31.598211    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dad794b4-7rtj8"
Sep 11 21:45:31.677154 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:31.677120    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24515852-1d43-491f-a428-3282e7bac38d-maas-api-service-ca\") pod \"24515852-1d43-491f-a428-3282e7bac38d\" (UID: \"24515852-1d43-491f-a428-3282e7bac38d\") "
Sep 11 21:45:31.677543 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:31.677505    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24515852-1d43-491f-a428-3282e7bac38d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "24515852-1d43-491f-a428-3282e7bac38d" (UID: "24515852-1d43-491f-a428-3282e7bac38d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:45:31.778722 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:31.778686    2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24515852-1d43-491f-a428-3282e7bac38d-maas-api-service-ca\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:32.465360 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:32.465327    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dad794b4-7rtj8"
Sep 11 21:45:32.465563 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:32.465362    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dad794b4-7rtj8" event={"ID":"24515852-1d43-491f-a428-3282e7bac38d","Type":"ContainerDied","Data":"636b8f3edf89a50d1fbd0947bc8b0d2ec48cbf11f4a1d6ad4784b14aa6d36467"}
Sep 11 21:45:32.465563 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:32.465415    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="636b8f3edf89a50d1fbd0947bc8b0d2ec48cbf11f4a1d6ad4784b14aa6d36467"
Sep 11 21:45:34.840859 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:34.840827    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr"]
Sep 11 21:45:34.841286 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:34.841030    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr" podUID="ade1c035-2822-4b9a-9222-84c59a32c668" containerName="payload-processing" containerID="cri-o://2449a85a68b651a2b303903cdf62d4a2391148c85c19d6339f73055dd5b40f57" gracePeriod=30
Sep 11 21:45:34.847709 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:34.847678    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d"]
Sep 11 21:45:34.847937 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:34.847910    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d" podUID="3629dec8-2b67-460b-96d7-21e5d2e4f6ea" containerName="payload-pre-processing" containerID="cri-o://98e4d68cf1b3508362662748c6e38d845dac31f8217373d9d223d312ae256724" gracePeriod=30
Sep 11 21:45:35.024855 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.024815    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dad794b4-7rtj8"]
Sep 11 21:45:35.029835 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.029802    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dad794b4-7rtj8"]
Sep 11 21:45:35.117703 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.117679    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr"
Sep 11 21:45:35.120494 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.120476    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d"
Sep 11 21:45:35.215803 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.215765    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4wn6\" (UniqueName: \"kubernetes.io/projected/ade1c035-2822-4b9a-9222-84c59a32c668-kube-api-access-q4wn6\") pod \"ade1c035-2822-4b9a-9222-84c59a32c668\" (UID: \"ade1c035-2822-4b9a-9222-84c59a32c668\") "
Sep 11 21:45:35.215972 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.215817    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46bh9\" (UniqueName: \"kubernetes.io/projected/3629dec8-2b67-460b-96d7-21e5d2e4f6ea-kube-api-access-46bh9\") pod \"3629dec8-2b67-460b-96d7-21e5d2e4f6ea\" (UID: \"3629dec8-2b67-460b-96d7-21e5d2e4f6ea\") "
Sep 11 21:45:35.216048 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.215999    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3629dec8-2b67-460b-96d7-21e5d2e4f6ea-plugins-config-volume\") pod \"3629dec8-2b67-460b-96d7-21e5d2e4f6ea\" (UID: \"3629dec8-2b67-460b-96d7-21e5d2e4f6ea\") "
Sep 11 21:45:35.216124 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.216107    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade1c035-2822-4b9a-9222-84c59a32c668-plugins-config-volume\") pod \"ade1c035-2822-4b9a-9222-84c59a32c668\" (UID: \"ade1c035-2822-4b9a-9222-84c59a32c668\") "
Sep 11 21:45:35.216350 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.216325    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3629dec8-2b67-460b-96d7-21e5d2e4f6ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3629dec8-2b67-460b-96d7-21e5d2e4f6ea" (UID: "3629dec8-2b67-460b-96d7-21e5d2e4f6ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:45:35.216490 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.216467    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade1c035-2822-4b9a-9222-84c59a32c668-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ade1c035-2822-4b9a-9222-84c59a32c668" (UID: "ade1c035-2822-4b9a-9222-84c59a32c668"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:45:35.216544 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.216501    2544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3629dec8-2b67-460b-96d7-21e5d2e4f6ea-plugins-config-volume\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:35.217939 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.217918    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3629dec8-2b67-460b-96d7-21e5d2e4f6ea-kube-api-access-46bh9" (OuterVolumeSpecName: "kube-api-access-46bh9") pod "3629dec8-2b67-460b-96d7-21e5d2e4f6ea" (UID: "3629dec8-2b67-460b-96d7-21e5d2e4f6ea"). InnerVolumeSpecName "kube-api-access-46bh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:45:35.218040 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.217951    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade1c035-2822-4b9a-9222-84c59a32c668-kube-api-access-q4wn6" (OuterVolumeSpecName: "kube-api-access-q4wn6") pod "ade1c035-2822-4b9a-9222-84c59a32c668" (UID: "ade1c035-2822-4b9a-9222-84c59a32c668"). InnerVolumeSpecName "kube-api-access-q4wn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:45:35.317442 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.317394    2544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade1c035-2822-4b9a-9222-84c59a32c668-plugins-config-volume\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:35.317442 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.317435    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4wn6\" (UniqueName: \"kubernetes.io/projected/ade1c035-2822-4b9a-9222-84c59a32c668-kube-api-access-q4wn6\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:35.317442 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.317449    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46bh9\" (UniqueName: \"kubernetes.io/projected/3629dec8-2b67-460b-96d7-21e5d2e4f6ea-kube-api-access-46bh9\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:35.485671 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.485636    2544 generic.go:356] "Generic (PLEG): container finished" podID="3629dec8-2b67-460b-96d7-21e5d2e4f6ea" containerID="98e4d68cf1b3508362662748c6e38d845dac31f8217373d9d223d312ae256724" exitCode=0
Sep 11 21:45:35.485671 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.485675    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d"
Sep 11 21:45:35.485945 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.485721    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d" event={"ID":"3629dec8-2b67-460b-96d7-21e5d2e4f6ea","Type":"ContainerDied","Data":"98e4d68cf1b3508362662748c6e38d845dac31f8217373d9d223d312ae256724"}
Sep 11 21:45:35.485945 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.485782    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d" event={"ID":"3629dec8-2b67-460b-96d7-21e5d2e4f6ea","Type":"ContainerDied","Data":"700b969fe26a55502ab3d1d20410fe6b58e3e1a3361ba2c06807ec731aa9ebde"}
Sep 11 21:45:35.485945 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.485816    2544 scope.go:117] "RemoveContainer" containerID="98e4d68cf1b3508362662748c6e38d845dac31f8217373d9d223d312ae256724"
Sep 11 21:45:35.494164 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.494133    2544 generic.go:356] "Generic (PLEG): container finished" podID="ade1c035-2822-4b9a-9222-84c59a32c668" containerID="2449a85a68b651a2b303903cdf62d4a2391148c85c19d6339f73055dd5b40f57" exitCode=0
Sep 11 21:45:35.494331 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.494200    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr" event={"ID":"ade1c035-2822-4b9a-9222-84c59a32c668","Type":"ContainerDied","Data":"2449a85a68b651a2b303903cdf62d4a2391148c85c19d6339f73055dd5b40f57"}
Sep 11 21:45:35.494331 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.494204    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr"
Sep 11 21:45:35.494331 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.494290    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr" event={"ID":"ade1c035-2822-4b9a-9222-84c59a32c668","Type":"ContainerDied","Data":"be543ea2d67b59b1c9c63eaa0a9ef7f8930a2f27feb2986a10ad65f666fd2759"}
Sep 11 21:45:35.503592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.503566    2544 scope.go:117] "RemoveContainer" containerID="98e4d68cf1b3508362662748c6e38d845dac31f8217373d9d223d312ae256724"
Sep 11 21:45:35.503851 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:45:35.503832    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98e4d68cf1b3508362662748c6e38d845dac31f8217373d9d223d312ae256724\": container with ID starting with 98e4d68cf1b3508362662748c6e38d845dac31f8217373d9d223d312ae256724 not found: ID does not exist" containerID="98e4d68cf1b3508362662748c6e38d845dac31f8217373d9d223d312ae256724"
Sep 11 21:45:35.503932 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.503863    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e4d68cf1b3508362662748c6e38d845dac31f8217373d9d223d312ae256724"} err="failed to get container status \"98e4d68cf1b3508362662748c6e38d845dac31f8217373d9d223d312ae256724\": rpc error: code = NotFound desc = could not find container \"98e4d68cf1b3508362662748c6e38d845dac31f8217373d9d223d312ae256724\": container with ID starting with 98e4d68cf1b3508362662748c6e38d845dac31f8217373d9d223d312ae256724 not found: ID does not exist"
Sep 11 21:45:35.503932 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.503887    2544 scope.go:117] "RemoveContainer" containerID="2449a85a68b651a2b303903cdf62d4a2391148c85c19d6339f73055dd5b40f57"
Sep 11 21:45:35.511955 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.511936    2544 scope.go:117] "RemoveContainer" containerID="2449a85a68b651a2b303903cdf62d4a2391148c85c19d6339f73055dd5b40f57"
Sep 11 21:45:35.512194 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:45:35.512175    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2449a85a68b651a2b303903cdf62d4a2391148c85c19d6339f73055dd5b40f57\": container with ID starting with 2449a85a68b651a2b303903cdf62d4a2391148c85c19d6339f73055dd5b40f57 not found: ID does not exist" containerID="2449a85a68b651a2b303903cdf62d4a2391148c85c19d6339f73055dd5b40f57"
Sep 11 21:45:35.512299 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.512201    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2449a85a68b651a2b303903cdf62d4a2391148c85c19d6339f73055dd5b40f57"} err="failed to get container status \"2449a85a68b651a2b303903cdf62d4a2391148c85c19d6339f73055dd5b40f57\": rpc error: code = NotFound desc = could not find container \"2449a85a68b651a2b303903cdf62d4a2391148c85c19d6339f73055dd5b40f57\": container with ID starting with 2449a85a68b651a2b303903cdf62d4a2391148c85c19d6339f73055dd5b40f57 not found: ID does not exist"
Sep 11 21:45:35.515119 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.515100    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d"]
Sep 11 21:45:35.519118 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.519095    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dad794b4-6688fbf6f8-jls7d"]
Sep 11 21:45:35.529475 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.529454    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr"]
Sep 11 21:45:35.533135 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.533114    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dad794b4-b69fbbc57-zzlnr"]
Sep 11 21:45:35.567230 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.567184    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:35.567408 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.567256    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:35.579888 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.579860    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:35.592076 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.592045    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24515852-1d43-491f-a428-3282e7bac38d" path="/var/lib/kubelet/pods/24515852-1d43-491f-a428-3282e7bac38d/volumes"
Sep 11 21:45:35.592533 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.592512    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3629dec8-2b67-460b-96d7-21e5d2e4f6ea" path="/var/lib/kubelet/pods/3629dec8-2b67-460b-96d7-21e5d2e4f6ea/volumes"
Sep 11 21:45:35.592912 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:35.592894    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ade1c035-2822-4b9a-9222-84c59a32c668" path="/var/lib/kubelet/pods/ade1c035-2822-4b9a-9222-84c59a32c668/volumes"
Sep 11 21:45:36.165841 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:36.165763    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"]
Sep 11 21:45:36.166213 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:36.165954    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l" podUID="bf1995f0-8282-4fee-b993-e9255506345e" containerName="istio-proxy" containerID="cri-o://fef65d5b933945664f32bbeed585106feb9ece82709eab0659697b5a0711557b" gracePeriod=30
Sep 11 21:45:36.512915 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:36.512885    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:45:41.436205 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.436143    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:45:41.475273 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.475243    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-istio-data\") pod \"bf1995f0-8282-4fee-b993-e9255506345e\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") "
Sep 11 21:45:41.475447 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.475281    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-workload-certs\") pod \"bf1995f0-8282-4fee-b993-e9255506345e\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") "
Sep 11 21:45:41.475447 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.475321    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1995f0-8282-4fee-b993-e9255506345e-istio-podinfo\") pod \"bf1995f0-8282-4fee-b993-e9255506345e\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") "
Sep 11 21:45:41.475447 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.475360    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwlhm\" (UniqueName: \"kubernetes.io/projected/bf1995f0-8282-4fee-b993-e9255506345e-kube-api-access-gwlhm\") pod \"bf1995f0-8282-4fee-b993-e9255506345e\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") "
Sep 11 21:45:41.475447 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.475388    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1995f0-8282-4fee-b993-e9255506345e-istiod-ca-cert\") pod \"bf1995f0-8282-4fee-b993-e9255506345e\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") "
Sep 11 21:45:41.475447 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.475440    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-workload-socket\") pod \"bf1995f0-8282-4fee-b993-e9255506345e\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") "
Sep 11 21:45:41.475721 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.475509    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-istio-envoy\") pod \"bf1995f0-8282-4fee-b993-e9255506345e\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") "
Sep 11 21:45:41.475721 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.475539    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf1995f0-8282-4fee-b993-e9255506345e-istio-token\") pod \"bf1995f0-8282-4fee-b993-e9255506345e\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") "
Sep 11 21:45:41.475721 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.475550    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-istio-data" (OuterVolumeSpecName: "istio-data") pod "bf1995f0-8282-4fee-b993-e9255506345e" (UID: "bf1995f0-8282-4fee-b993-e9255506345e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:45:41.475721 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.475573    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-credential-socket\") pod \"bf1995f0-8282-4fee-b993-e9255506345e\" (UID: \"bf1995f0-8282-4fee-b993-e9255506345e\") "
Sep 11 21:45:41.475721 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.475656    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bf1995f0-8282-4fee-b993-e9255506345e" (UID: "bf1995f0-8282-4fee-b993-e9255506345e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:45:41.475994 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.475879    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bf1995f0-8282-4fee-b993-e9255506345e" (UID: "bf1995f0-8282-4fee-b993-e9255506345e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:45:41.475994 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.475934    2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-istio-data\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:41.475994 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.475953    2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-workload-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:41.476156 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.476007    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf1995f0-8282-4fee-b993-e9255506345e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bf1995f0-8282-4fee-b993-e9255506345e" (UID: "bf1995f0-8282-4fee-b993-e9255506345e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:45:41.476156 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.476117    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bf1995f0-8282-4fee-b993-e9255506345e" (UID: "bf1995f0-8282-4fee-b993-e9255506345e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:45:41.477720 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.477689    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf1995f0-8282-4fee-b993-e9255506345e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bf1995f0-8282-4fee-b993-e9255506345e" (UID: "bf1995f0-8282-4fee-b993-e9255506345e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:45:41.477720 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.477688    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bf1995f0-8282-4fee-b993-e9255506345e" (UID: "bf1995f0-8282-4fee-b993-e9255506345e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:45:41.477890 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.477770    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1995f0-8282-4fee-b993-e9255506345e-kube-api-access-gwlhm" (OuterVolumeSpecName: "kube-api-access-gwlhm") pod "bf1995f0-8282-4fee-b993-e9255506345e" (UID: "bf1995f0-8282-4fee-b993-e9255506345e"). InnerVolumeSpecName "kube-api-access-gwlhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:45:41.478066 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.478050    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1995f0-8282-4fee-b993-e9255506345e-istio-token" (OuterVolumeSpecName: "istio-token") pod "bf1995f0-8282-4fee-b993-e9255506345e" (UID: "bf1995f0-8282-4fee-b993-e9255506345e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:45:41.532096 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.532059    2544 generic.go:356] "Generic (PLEG): container finished" podID="bf1995f0-8282-4fee-b993-e9255506345e" containerID="fef65d5b933945664f32bbeed585106feb9ece82709eab0659697b5a0711557b" exitCode=0
Sep 11 21:45:41.532283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.532146    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"
Sep 11 21:45:41.532283 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.532144    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l" event={"ID":"bf1995f0-8282-4fee-b993-e9255506345e","Type":"ContainerDied","Data":"fef65d5b933945664f32bbeed585106feb9ece82709eab0659697b5a0711557b"}
Sep 11 21:45:41.532377 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.532293    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l" event={"ID":"bf1995f0-8282-4fee-b993-e9255506345e","Type":"ContainerDied","Data":"df7ab3742103fdecfa90e83fe7f85eab6ab9621d6ca4a3e025638efb231da36e"}
Sep 11 21:45:41.532377 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.532315    2544 scope.go:117] "RemoveContainer" containerID="fef65d5b933945664f32bbeed585106feb9ece82709eab0659697b5a0711557b"
Sep 11 21:45:41.541482 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.541465    2544 scope.go:117] "RemoveContainer" containerID="eb491b2185dec23d6c20b268f07f949960b019cb661c4812714e0a13d689a204"
Sep 11 21:45:41.549361 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.549342    2544 scope.go:117] "RemoveContainer" containerID="fef65d5b933945664f32bbeed585106feb9ece82709eab0659697b5a0711557b"
Sep 11 21:45:41.549615 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:45:41.549594    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fef65d5b933945664f32bbeed585106feb9ece82709eab0659697b5a0711557b\": container with ID starting with fef65d5b933945664f32bbeed585106feb9ece82709eab0659697b5a0711557b not found: ID does not exist" containerID="fef65d5b933945664f32bbeed585106feb9ece82709eab0659697b5a0711557b"
Sep 11 21:45:41.549658 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.549625    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef65d5b933945664f32bbeed585106feb9ece82709eab0659697b5a0711557b"} err="failed to get container status \"fef65d5b933945664f32bbeed585106feb9ece82709eab0659697b5a0711557b\": rpc error: code = NotFound desc = could not find container \"fef65d5b933945664f32bbeed585106feb9ece82709eab0659697b5a0711557b\": container with ID starting with fef65d5b933945664f32bbeed585106feb9ece82709eab0659697b5a0711557b not found: ID does not exist"
Sep 11 21:45:41.549658 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.549641    2544 scope.go:117] "RemoveContainer" containerID="eb491b2185dec23d6c20b268f07f949960b019cb661c4812714e0a13d689a204"
Sep 11 21:45:41.549893 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:45:41.549879    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb491b2185dec23d6c20b268f07f949960b019cb661c4812714e0a13d689a204\": container with ID starting with eb491b2185dec23d6c20b268f07f949960b019cb661c4812714e0a13d689a204 not found: ID does not exist" containerID="eb491b2185dec23d6c20b268f07f949960b019cb661c4812714e0a13d689a204"
Sep 11 21:45:41.549934 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.549897    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb491b2185dec23d6c20b268f07f949960b019cb661c4812714e0a13d689a204"} err="failed to get container status \"eb491b2185dec23d6c20b268f07f949960b019cb661c4812714e0a13d689a204\": rpc error: code = NotFound desc = could not find container \"eb491b2185dec23d6c20b268f07f949960b019cb661c4812714e0a13d689a204\": container with ID starting with eb491b2185dec23d6c20b268f07f949960b019cb661c4812714e0a13d689a204 not found: ID does not exist"
Sep 11 21:45:41.553839 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.553816    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"]
Sep 11 21:45:41.557402 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.557383    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-dad794b4-openshift-default-67c58bfb86-m5d2l"]
Sep 11 21:45:41.576920 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.576896    2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-workload-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:41.576920 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.576920    2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-istio-envoy\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:41.577110 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.576931    2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf1995f0-8282-4fee-b993-e9255506345e-istio-token\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:41.577110 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.576939    2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1995f0-8282-4fee-b993-e9255506345e-credential-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:41.577110 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.576948    2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1995f0-8282-4fee-b993-e9255506345e-istio-podinfo\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:41.577110 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.576958    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwlhm\" (UniqueName: \"kubernetes.io/projected/bf1995f0-8282-4fee-b993-e9255506345e-kube-api-access-gwlhm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:41.577110 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.576972    2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1995f0-8282-4fee-b993-e9255506345e-istiod-ca-cert\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:45:41.592013 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:41.591986    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf1995f0-8282-4fee-b993-e9255506345e" path="/var/lib/kubelet/pods/bf1995f0-8282-4fee-b993-e9255506345e/volumes"
Sep 11 21:45:44.059966 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.059929    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj"]
Sep 11 21:45:44.060488 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.060449    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24515852-1d43-491f-a428-3282e7bac38d" containerName="revoke-keys"
Sep 11 21:45:44.060488 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.060471    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="24515852-1d43-491f-a428-3282e7bac38d" containerName="revoke-keys"
Sep 11 21:45:44.060606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.060491    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf1995f0-8282-4fee-b993-e9255506345e" containerName="istio-proxy"
Sep 11 21:45:44.060606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.060500    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf1995f0-8282-4fee-b993-e9255506345e" containerName="istio-proxy"
Sep 11 21:45:44.060606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.060511    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade1c035-2822-4b9a-9222-84c59a32c668" containerName="payload-processing"
Sep 11 21:45:44.060606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.060521    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade1c035-2822-4b9a-9222-84c59a32c668" containerName="payload-processing"
Sep 11 21:45:44.060606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.060550    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3629dec8-2b67-460b-96d7-21e5d2e4f6ea" containerName="payload-pre-processing"
Sep 11 21:45:44.060606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.060558    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="3629dec8-2b67-460b-96d7-21e5d2e4f6ea" containerName="payload-pre-processing"
Sep 11 21:45:44.060893 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.060645    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="3629dec8-2b67-460b-96d7-21e5d2e4f6ea" containerName="payload-pre-processing"
Sep 11 21:45:44.060893 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.060658    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf1995f0-8282-4fee-b993-e9255506345e" containerName="istio-proxy"
Sep 11 21:45:44.060893 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.060669    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf1995f0-8282-4fee-b993-e9255506345e" containerName="istio-proxy"
Sep 11 21:45:44.060893 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.060680    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="24515852-1d43-491f-a428-3282e7bac38d" containerName="revoke-keys"
Sep 11 21:45:44.060893 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.060692    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade1c035-2822-4b9a-9222-84c59a32c668" containerName="payload-processing"
Sep 11 21:45:44.064856 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.064834    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj"
Sep 11 21:45:44.067315 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.067283    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-7a9aa317\""
Sep 11 21:45:44.067453 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.067329    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-7a9aa317-dockercfg-lnb98\""
Sep 11 21:45:44.079106 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.079082    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj"]
Sep 11 21:45:44.092828 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.092800    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw"]
Sep 11 21:45:44.093143 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.093129    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf1995f0-8282-4fee-b993-e9255506345e" containerName="istio-proxy"
Sep 11 21:45:44.093143 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.093143    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf1995f0-8282-4fee-b993-e9255506345e" containerName="istio-proxy"
Sep 11 21:45:44.096281 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.096260    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw"
Sep 11 21:45:44.101656 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.101636    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjg7l\" (UniqueName: \"kubernetes.io/projected/a94bc216-b951-4421-af51-4a01a48389e1-kube-api-access-qjg7l\") pod \"payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj\" (UID: \"a94bc216-b951-4421-af51-4a01a48389e1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj"
Sep 11 21:45:44.101759 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.101696    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a94bc216-b951-4421-af51-4a01a48389e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj\" (UID: \"a94bc216-b951-4421-af51-4a01a48389e1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj"
Sep 11 21:45:44.106013 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.105992    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw"]
Sep 11 21:45:44.203050 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.203021    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbt9r\" (UniqueName: \"kubernetes.io/projected/f958c5bf-e52f-4857-902e-e82172d7e4ac-kube-api-access-kbt9r\") pod \"payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw\" (UID: \"f958c5bf-e52f-4857-902e-e82172d7e4ac\") " pod="openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw"
Sep 11 21:45:44.203248 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.203057    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjg7l\" (UniqueName: \"kubernetes.io/projected/a94bc216-b951-4421-af51-4a01a48389e1-kube-api-access-qjg7l\") pod \"payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj\" (UID: \"a94bc216-b951-4421-af51-4a01a48389e1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj"
Sep 11 21:45:44.203248 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.203140    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f958c5bf-e52f-4857-902e-e82172d7e4ac-plugins-config-volume\") pod \"payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw\" (UID: \"f958c5bf-e52f-4857-902e-e82172d7e4ac\") " pod="openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw"
Sep 11 21:45:44.203368 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.203240    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a94bc216-b951-4421-af51-4a01a48389e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj\" (UID: \"a94bc216-b951-4421-af51-4a01a48389e1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj"
Sep 11 21:45:44.203896 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.203871    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a94bc216-b951-4421-af51-4a01a48389e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj\" (UID: \"a94bc216-b951-4421-af51-4a01a48389e1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj"
Sep 11 21:45:44.211805 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.211780    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjg7l\" (UniqueName: \"kubernetes.io/projected/a94bc216-b951-4421-af51-4a01a48389e1-kube-api-access-qjg7l\") pod \"payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj\" (UID: \"a94bc216-b951-4421-af51-4a01a48389e1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj"
Sep 11 21:45:44.304623 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.304583    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbt9r\" (UniqueName: \"kubernetes.io/projected/f958c5bf-e52f-4857-902e-e82172d7e4ac-kube-api-access-kbt9r\") pod \"payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw\" (UID: \"f958c5bf-e52f-4857-902e-e82172d7e4ac\") " pod="openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw"
Sep 11 21:45:44.304797 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.304642    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f958c5bf-e52f-4857-902e-e82172d7e4ac-plugins-config-volume\") pod \"payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw\" (UID: \"f958c5bf-e52f-4857-902e-e82172d7e4ac\") " pod="openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw"
Sep 11 21:45:44.305442 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.305422    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f958c5bf-e52f-4857-902e-e82172d7e4ac-plugins-config-volume\") pod \"payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw\" (UID: \"f958c5bf-e52f-4857-902e-e82172d7e4ac\") " pod="openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw"
Sep 11 21:45:44.322846 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.322773    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbt9r\" (UniqueName: \"kubernetes.io/projected/f958c5bf-e52f-4857-902e-e82172d7e4ac-kube-api-access-kbt9r\") pod \"payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw\" (UID: \"f958c5bf-e52f-4857-902e-e82172d7e4ac\") " pod="openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw"
Sep 11 21:45:44.375237 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.375191    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj"
Sep 11 21:45:44.406094 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.406056    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw"
Sep 11 21:45:44.651589 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.650813    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj"]
Sep 11 21:45:44.686144 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:44.686118    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw"]
Sep 11 21:45:44.687590 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:45:44.687567    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf958c5bf_e52f_4857_902e_e82172d7e4ac.slice/crio-80eea866e227a51b083b4a04cbbc21eabb4af408a82fe8c844c0f46fa70f06d9 WatchSource:0}: Error finding container 80eea866e227a51b083b4a04cbbc21eabb4af408a82fe8c844c0f46fa70f06d9: Status 404 returned error can't find the container with id 80eea866e227a51b083b4a04cbbc21eabb4af408a82fe8c844c0f46fa70f06d9
Sep 11 21:45:45.555349 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:45.555310    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj" event={"ID":"a94bc216-b951-4421-af51-4a01a48389e1","Type":"ContainerStarted","Data":"a4986ef68b7336d2ad7630659587734ff99c00cd236ab3309d137bf8ae5f9879"}
Sep 11 21:45:45.555836 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:45.555375    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj"
Sep 11 21:45:45.555836 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:45.555399    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj" event={"ID":"a94bc216-b951-4421-af51-4a01a48389e1","Type":"ContainerStarted","Data":"a7a581f5391a675c7521631485ec4674ee00c8bb2c5f0e392778c70261f99b9d"}
Sep 11 21:45:45.556952 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:45.556920    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw" event={"ID":"f958c5bf-e52f-4857-902e-e82172d7e4ac","Type":"ContainerStarted","Data":"aab96f7cc0bca65569d6aa439de8be368f477bf2f0579de5764d40a5db3219cb"}
Sep 11 21:45:45.557084 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:45.556970    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw" event={"ID":"f958c5bf-e52f-4857-902e-e82172d7e4ac","Type":"ContainerStarted","Data":"80eea866e227a51b083b4a04cbbc21eabb4af408a82fe8c844c0f46fa70f06d9"}
Sep 11 21:45:45.557084 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:45.557010    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw"
Sep 11 21:45:45.572105 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:45.572048    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj" podStartSLOduration=1.572029846 podStartE2EDuration="1.572029846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:45:45.571256778 +0000 UTC m=+908.540673655" watchObservedRunningTime="2026-09-11 21:45:45.572029846 +0000 UTC m=+908.541446666"
Sep 11 21:45:45.588648 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:45.588599    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw" podStartSLOduration=1.588582733 podStartE2EDuration="1.588582733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:45:45.586248042 +0000 UTC m=+908.555664864" watchObservedRunningTime="2026-09-11 21:45:45.588582733 +0000 UTC m=+908.557999552"
Sep 11 21:45:56.564825 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:56.564791    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj"
Sep 11 21:45:56.565353 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:45:56.565029    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw"
Sep 11 21:46:00.047146 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:00.047107    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5b160c-rk9ff"]
Sep 11 21:46:00.054037 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:00.051970    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5b160c-rk9ff"
Sep 11 21:46:00.054887 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:00.054864    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 21:46:00.055003 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:00.054979    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:46:00.058273 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:00.058252    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5b160c-rk9ff"]
Sep 11 21:46:00.171596 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:00.171553    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1da732-b9fc-4f8b-8018-1fcfb341e4c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5b160c-rk9ff\" (UID: \"eb1da732-b9fc-4f8b-8018-1fcfb341e4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5b160c-rk9ff"
Sep 11 21:46:00.273038 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:00.272991    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1da732-b9fc-4f8b-8018-1fcfb341e4c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5b160c-rk9ff\" (UID: \"eb1da732-b9fc-4f8b-8018-1fcfb341e4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5b160c-rk9ff"
Sep 11 21:46:00.273684 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:00.273661    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1da732-b9fc-4f8b-8018-1fcfb341e4c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5b160c-rk9ff\" (UID: \"eb1da732-b9fc-4f8b-8018-1fcfb341e4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5b160c-rk9ff"
Sep 11 21:46:00.366528 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:00.366443    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5b160c-rk9ff"
Sep 11 21:46:00.636583 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:00.636552    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5b160c-rk9ff"]
Sep 11 21:46:00.639187 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:46:00.639161    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1da732_b9fc_4f8b_8018_1fcfb341e4c4.slice/crio-ed70c8b3e484b08ddb0585f212c4579701a3632ff58ae5d447034579e634e68d WatchSource:0}: Error finding container ed70c8b3e484b08ddb0585f212c4579701a3632ff58ae5d447034579e634e68d: Status 404 returned error can't find the container with id ed70c8b3e484b08ddb0585f212c4579701a3632ff58ae5d447034579e634e68d
Sep 11 21:46:01.643937 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:01.643842    2544 generic.go:356] "Generic (PLEG): container finished" podID="eb1da732-b9fc-4f8b-8018-1fcfb341e4c4" containerID="e0b6b92d57f4772d0aab6b197df8341dac0e710e58c19f79f350af14c91a835b" exitCode=0
Sep 11 21:46:01.643937 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:01.643894    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5b160c-rk9ff" event={"ID":"eb1da732-b9fc-4f8b-8018-1fcfb341e4c4","Type":"ContainerDied","Data":"e0b6b92d57f4772d0aab6b197df8341dac0e710e58c19f79f350af14c91a835b"}
Sep 11 21:46:01.643937 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:01.643936    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5b160c-rk9ff" event={"ID":"eb1da732-b9fc-4f8b-8018-1fcfb341e4c4","Type":"ContainerStarted","Data":"ed70c8b3e484b08ddb0585f212c4579701a3632ff58ae5d447034579e634e68d"}
Sep 11 21:46:02.790503 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:02.790473    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5b160c-rk9ff"
Sep 11 21:46:02.898140 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:02.898104    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1da732-b9fc-4f8b-8018-1fcfb341e4c4-maas-api-service-ca\") pod \"eb1da732-b9fc-4f8b-8018-1fcfb341e4c4\" (UID: \"eb1da732-b9fc-4f8b-8018-1fcfb341e4c4\") "
Sep 11 21:46:02.898454 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:02.898430    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1da732-b9fc-4f8b-8018-1fcfb341e4c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb1da732-b9fc-4f8b-8018-1fcfb341e4c4" (UID: "eb1da732-b9fc-4f8b-8018-1fcfb341e4c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:46:02.999845 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:02.999810    2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1da732-b9fc-4f8b-8018-1fcfb341e4c4-maas-api-service-ca\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:46:03.566332 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.566297    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"]
Sep 11 21:46:03.566674 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.566663    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb1da732-b9fc-4f8b-8018-1fcfb341e4c4" containerName="revoke-keys"
Sep 11 21:46:03.566732 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.566676    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb1da732-b9fc-4f8b-8018-1fcfb341e4c4" containerName="revoke-keys"
Sep 11 21:46:03.566769 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.566732    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb1da732-b9fc-4f8b-8018-1fcfb341e4c4" containerName="revoke-keys"
Sep 11 21:46:03.569802 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.569786    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.573334 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.573302    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f12741\"/\"openshift-service-ca.crt\""
Sep 11 21:46:03.573496 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.573301    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f12741\"/\"simulator-w0-f12741-kserve-self-signed-certs\""
Sep 11 21:46:03.573496 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.573383    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f12741\"/\"default-dockercfg-9ztcv\""
Sep 11 21:46:03.573496 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.573309    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f12741\"/\"kube-root-ca.crt\""
Sep 11 21:46:03.579501 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.579481    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"]
Sep 11 21:46:03.655881 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.655855    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5b160c-rk9ff"
Sep 11 21:46:03.656039 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.655854    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5b160c-rk9ff" event={"ID":"eb1da732-b9fc-4f8b-8018-1fcfb341e4c4","Type":"ContainerDied","Data":"ed70c8b3e484b08ddb0585f212c4579701a3632ff58ae5d447034579e634e68d"}
Sep 11 21:46:03.656039 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.655980    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed70c8b3e484b08ddb0585f212c4579701a3632ff58ae5d447034579e634e68d"
Sep 11 21:46:03.706723 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.706687    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-dshm\") pod \"simulator-w0-f12741-kserve-f9895bb74-j57kq\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") " pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.706918 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.706733    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrslb\" (UniqueName: \"kubernetes.io/projected/56ec433b-c8ea-44a6-8f0b-1032239369f7-kube-api-access-qrslb\") pod \"simulator-w0-f12741-kserve-f9895bb74-j57kq\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") " pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.706918 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.706800    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-tmp-dir\") pod \"simulator-w0-f12741-kserve-f9895bb74-j57kq\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") " pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.706918 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.706816    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ec433b-c8ea-44a6-8f0b-1032239369f7-tls-certs\") pod \"simulator-w0-f12741-kserve-f9895bb74-j57kq\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") " pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.706918 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.706835    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-model-cache\") pod \"simulator-w0-f12741-kserve-f9895bb74-j57kq\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") " pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.706918 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.706857    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-home\") pod \"simulator-w0-f12741-kserve-f9895bb74-j57kq\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") " pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.807633 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.807599    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-tmp-dir\") pod \"simulator-w0-f12741-kserve-f9895bb74-j57kq\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") " pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.807633 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.807637    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ec433b-c8ea-44a6-8f0b-1032239369f7-tls-certs\") pod \"simulator-w0-f12741-kserve-f9895bb74-j57kq\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") " pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.808201 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.807709    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-model-cache\") pod \"simulator-w0-f12741-kserve-f9895bb74-j57kq\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") " pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.808201 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.807759    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-home\") pod \"simulator-w0-f12741-kserve-f9895bb74-j57kq\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") " pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.808201 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.807871    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-dshm\") pod \"simulator-w0-f12741-kserve-f9895bb74-j57kq\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") " pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.808201 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.807911    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrslb\" (UniqueName: \"kubernetes.io/projected/56ec433b-c8ea-44a6-8f0b-1032239369f7-kube-api-access-qrslb\") pod \"simulator-w0-f12741-kserve-f9895bb74-j57kq\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") " pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.808406 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.808265    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-model-cache\") pod \"simulator-w0-f12741-kserve-f9895bb74-j57kq\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") " pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.808406 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.808354    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-home\") pod \"simulator-w0-f12741-kserve-f9895bb74-j57kq\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") " pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.808406 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.808392    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-tmp-dir\") pod \"simulator-w0-f12741-kserve-f9895bb74-j57kq\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") " pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.810059 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.810036    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-dshm\") pod \"simulator-w0-f12741-kserve-f9895bb74-j57kq\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") " pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.810488 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.810467    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ec433b-c8ea-44a6-8f0b-1032239369f7-tls-certs\") pod \"simulator-w0-f12741-kserve-f9895bb74-j57kq\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") " pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.817091 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.817027    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrslb\" (UniqueName: \"kubernetes.io/projected/56ec433b-c8ea-44a6-8f0b-1032239369f7-kube-api-access-qrslb\") pod \"simulator-w0-f12741-kserve-f9895bb74-j57kq\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") " pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:03.879628 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:03.879587    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:04.155145 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:04.155114    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"]
Sep 11 21:46:04.157075 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:46:04.157047    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56ec433b_c8ea_44a6_8f0b_1032239369f7.slice/crio-ebacbb1a09ee389c17d9a4603fc7e5daab8993fdefe3678ee181f30f124e0f23 WatchSource:0}: Error finding container ebacbb1a09ee389c17d9a4603fc7e5daab8993fdefe3678ee181f30f124e0f23: Status 404 returned error can't find the container with id ebacbb1a09ee389c17d9a4603fc7e5daab8993fdefe3678ee181f30f124e0f23
Sep 11 21:46:04.667884 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:04.667840    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq" event={"ID":"56ec433b-c8ea-44a6-8f0b-1032239369f7","Type":"ContainerStarted","Data":"8098522d20d28906e64aa9f19b7330b705a384e8d5cf6d07ffd1cc91aa27add9"}
Sep 11 21:46:04.668091 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:04.667893    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq" event={"ID":"56ec433b-c8ea-44a6-8f0b-1032239369f7","Type":"ContainerStarted","Data":"ebacbb1a09ee389c17d9a4603fc7e5daab8993fdefe3678ee181f30f124e0f23"}
Sep 11 21:46:04.690887 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:04.690821    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq" podStartSLOduration=1.690805153 podStartE2EDuration="1.690805153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:46:04.689468086 +0000 UTC m=+927.658884907" watchObservedRunningTime="2026-09-11 21:46:04.690805153 +0000 UTC m=+927.660221972"
Sep 11 21:46:09.500480 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:09.500439    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t"]
Sep 11 21:46:09.503959 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:09.503943    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t"
Sep 11 21:46:09.506424 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:09.506403    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:46:09.506533 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:09.506404    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 21:46:09.510491 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:09.510462    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t"]
Sep 11 21:46:09.562342 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:09.562300    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d4f7fc-5d9b-402f-8959-6a027a4ed533-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t\" (UID: \"b7d4f7fc-5d9b-402f-8959-6a027a4ed533\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t"
Sep 11 21:46:09.663656 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:09.663611    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d4f7fc-5d9b-402f-8959-6a027a4ed533-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t\" (UID: \"b7d4f7fc-5d9b-402f-8959-6a027a4ed533\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t"
Sep 11 21:46:09.664229 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:09.664197    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d4f7fc-5d9b-402f-8959-6a027a4ed533-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t\" (UID: \"b7d4f7fc-5d9b-402f-8959-6a027a4ed533\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t"
Sep 11 21:46:09.816384 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:09.816296    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t"
Sep 11 21:46:10.089388 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:10.089358    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t"]
Sep 11 21:46:10.090963 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:46:10.090931    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d4f7fc_5d9b_402f_8959_6a027a4ed533.slice/crio-f260b3f22d57b1c76b6ac0b7b13039b956fbbe8ee70dabb366f81a31f7b796a0 WatchSource:0}: Error finding container f260b3f22d57b1c76b6ac0b7b13039b956fbbe8ee70dabb366f81a31f7b796a0: Status 404 returned error can't find the container with id f260b3f22d57b1c76b6ac0b7b13039b956fbbe8ee70dabb366f81a31f7b796a0
Sep 11 21:46:10.708494 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:10.708407    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t" event={"ID":"b7d4f7fc-5d9b-402f-8959-6a027a4ed533","Type":"ContainerStarted","Data":"72d5b3eabb36231e76e75956c5a815ecb457feee93af9e2b4f03e10fabc3eca6"}
Sep 11 21:46:10.708494 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:10.708454    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t" event={"ID":"b7d4f7fc-5d9b-402f-8959-6a027a4ed533","Type":"ContainerStarted","Data":"f260b3f22d57b1c76b6ac0b7b13039b956fbbe8ee70dabb366f81a31f7b796a0"}
Sep 11 21:46:10.723787 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:10.723737    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t" podStartSLOduration=1.723720951 podStartE2EDuration="1.723720951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:46:10.722074914 +0000 UTC m=+933.691491734" watchObservedRunningTime="2026-09-11 21:46:10.723720951 +0000 UTC m=+933.693137820"
Sep 11 21:46:11.730240 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:11.729838    2544 generic.go:356] "Generic (PLEG): container finished" podID="b7d4f7fc-5d9b-402f-8959-6a027a4ed533" containerID="72d5b3eabb36231e76e75956c5a815ecb457feee93af9e2b4f03e10fabc3eca6" exitCode=0
Sep 11 21:46:11.730240 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:11.730003    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t" event={"ID":"b7d4f7fc-5d9b-402f-8959-6a027a4ed533","Type":"ContainerDied","Data":"72d5b3eabb36231e76e75956c5a815ecb457feee93af9e2b4f03e10fabc3eca6"}
Sep 11 21:46:12.060368 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:12.060261    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5b160c-rk9ff"]
Sep 11 21:46:12.062382 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:12.062358    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5b160c-rk9ff"]
Sep 11 21:46:12.873269 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:12.873245    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t"
Sep 11 21:46:12.895528 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:12.895491    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d4f7fc-5d9b-402f-8959-6a027a4ed533-maas-api-service-ca\") pod \"b7d4f7fc-5d9b-402f-8959-6a027a4ed533\" (UID: \"b7d4f7fc-5d9b-402f-8959-6a027a4ed533\") "
Sep 11 21:46:12.895892 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:12.895866    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d4f7fc-5d9b-402f-8959-6a027a4ed533-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b7d4f7fc-5d9b-402f-8959-6a027a4ed533" (UID: "b7d4f7fc-5d9b-402f-8959-6a027a4ed533"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:46:12.997180 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:12.997084    2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d4f7fc-5d9b-402f-8959-6a027a4ed533-maas-api-service-ca\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:46:13.596354 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:13.593417    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb1da732-b9fc-4f8b-8018-1fcfb341e4c4" path="/var/lib/kubelet/pods/eb1da732-b9fc-4f8b-8018-1fcfb341e4c4/volumes"
Sep 11 21:46:13.742683 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:13.742654    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t"
Sep 11 21:46:13.742683 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:13.742663    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t" event={"ID":"b7d4f7fc-5d9b-402f-8959-6a027a4ed533","Type":"ContainerDied","Data":"f260b3f22d57b1c76b6ac0b7b13039b956fbbe8ee70dabb366f81a31f7b796a0"}
Sep 11 21:46:13.742889 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:13.742709    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f260b3f22d57b1c76b6ac0b7b13039b956fbbe8ee70dabb366f81a31f7b796a0"
Sep 11 21:46:13.880455 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:13.880366    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:13.880455 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:13.880413    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:13.893195 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:13.893168    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:14.759303 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:14.759265    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:46:17.412086 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:17.412051    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"]
Sep 11 21:46:21.359515 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.359481    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw"]
Sep 11 21:46:21.359979 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.359690    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw" podUID="f958c5bf-e52f-4857-902e-e82172d7e4ac" containerName="payload-processing" containerID="cri-o://aab96f7cc0bca65569d6aa439de8be368f477bf2f0579de5764d40a5db3219cb" gracePeriod=30
Sep 11 21:46:21.367338 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.367044    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj"]
Sep 11 21:46:21.367338 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.367285    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj" podUID="a94bc216-b951-4421-af51-4a01a48389e1" containerName="payload-pre-processing" containerID="cri-o://a4986ef68b7336d2ad7630659587734ff99c00cd236ab3309d137bf8ae5f9879" gracePeriod=30
Sep 11 21:46:21.535815 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.535775    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t"]
Sep 11 21:46:21.541379 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.541347    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a9aa317-4wf7t"]
Sep 11 21:46:21.594311 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.594263    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7d4f7fc-5d9b-402f-8959-6a027a4ed533" path="/var/lib/kubelet/pods/b7d4f7fc-5d9b-402f-8959-6a027a4ed533/volumes"
Sep 11 21:46:21.680018 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.679986    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw"
Sep 11 21:46:21.683068 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.683045    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj"
Sep 11 21:46:21.786550 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.786506    2544 generic.go:356] "Generic (PLEG): container finished" podID="a94bc216-b951-4421-af51-4a01a48389e1" containerID="a4986ef68b7336d2ad7630659587734ff99c00cd236ab3309d137bf8ae5f9879" exitCode=0
Sep 11 21:46:21.786858 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.786562    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj"
Sep 11 21:46:21.786992 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.786580    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj" event={"ID":"a94bc216-b951-4421-af51-4a01a48389e1","Type":"ContainerDied","Data":"a4986ef68b7336d2ad7630659587734ff99c00cd236ab3309d137bf8ae5f9879"}
Sep 11 21:46:21.787066 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.787042    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj" event={"ID":"a94bc216-b951-4421-af51-4a01a48389e1","Type":"ContainerDied","Data":"a7a581f5391a675c7521631485ec4674ee00c8bb2c5f0e392778c70261f99b9d"}
Sep 11 21:46:21.787122 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.787100    2544 scope.go:117] "RemoveContainer" containerID="a4986ef68b7336d2ad7630659587734ff99c00cd236ab3309d137bf8ae5f9879"
Sep 11 21:46:21.789849 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.789657    2544 generic.go:356] "Generic (PLEG): container finished" podID="f958c5bf-e52f-4857-902e-e82172d7e4ac" containerID="aab96f7cc0bca65569d6aa439de8be368f477bf2f0579de5764d40a5db3219cb" exitCode=0
Sep 11 21:46:21.789849 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.789701    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw"
Sep 11 21:46:21.789849 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.789772    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw" event={"ID":"f958c5bf-e52f-4857-902e-e82172d7e4ac","Type":"ContainerDied","Data":"aab96f7cc0bca65569d6aa439de8be368f477bf2f0579de5764d40a5db3219cb"}
Sep 11 21:46:21.789849 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.789814    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw" event={"ID":"f958c5bf-e52f-4857-902e-e82172d7e4ac","Type":"ContainerDied","Data":"80eea866e227a51b083b4a04cbbc21eabb4af408a82fe8c844c0f46fa70f06d9"}
Sep 11 21:46:21.792583 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.792559    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a94bc216-b951-4421-af51-4a01a48389e1-plugins-config-volume\") pod \"a94bc216-b951-4421-af51-4a01a48389e1\" (UID: \"a94bc216-b951-4421-af51-4a01a48389e1\") "
Sep 11 21:46:21.792726 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.792710    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbt9r\" (UniqueName: \"kubernetes.io/projected/f958c5bf-e52f-4857-902e-e82172d7e4ac-kube-api-access-kbt9r\") pod \"f958c5bf-e52f-4857-902e-e82172d7e4ac\" (UID: \"f958c5bf-e52f-4857-902e-e82172d7e4ac\") "
Sep 11 21:46:21.792798 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.792764    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjg7l\" (UniqueName: \"kubernetes.io/projected/a94bc216-b951-4421-af51-4a01a48389e1-kube-api-access-qjg7l\") pod \"a94bc216-b951-4421-af51-4a01a48389e1\" (UID: \"a94bc216-b951-4421-af51-4a01a48389e1\") "
Sep 11 21:46:21.792798 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.792789    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f958c5bf-e52f-4857-902e-e82172d7e4ac-plugins-config-volume\") pod \"f958c5bf-e52f-4857-902e-e82172d7e4ac\" (UID: \"f958c5bf-e52f-4857-902e-e82172d7e4ac\") "
Sep 11 21:46:21.793344 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.793037    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94bc216-b951-4421-af51-4a01a48389e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a94bc216-b951-4421-af51-4a01a48389e1" (UID: "a94bc216-b951-4421-af51-4a01a48389e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:46:21.793449 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.793416    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f958c5bf-e52f-4857-902e-e82172d7e4ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f958c5bf-e52f-4857-902e-e82172d7e4ac" (UID: "f958c5bf-e52f-4857-902e-e82172d7e4ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:46:21.793536 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.793515    2544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f958c5bf-e52f-4857-902e-e82172d7e4ac-plugins-config-volume\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:46:21.793607 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.793542    2544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a94bc216-b951-4421-af51-4a01a48389e1-plugins-config-volume\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:46:21.795113 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.795090    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94bc216-b951-4421-af51-4a01a48389e1-kube-api-access-qjg7l" (OuterVolumeSpecName: "kube-api-access-qjg7l") pod "a94bc216-b951-4421-af51-4a01a48389e1" (UID: "a94bc216-b951-4421-af51-4a01a48389e1"). InnerVolumeSpecName "kube-api-access-qjg7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:46:21.795285 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.795207    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f958c5bf-e52f-4857-902e-e82172d7e4ac-kube-api-access-kbt9r" (OuterVolumeSpecName: "kube-api-access-kbt9r") pod "f958c5bf-e52f-4857-902e-e82172d7e4ac" (UID: "f958c5bf-e52f-4857-902e-e82172d7e4ac"). InnerVolumeSpecName "kube-api-access-kbt9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:46:21.798352 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.798331    2544 scope.go:117] "RemoveContainer" containerID="a4986ef68b7336d2ad7630659587734ff99c00cd236ab3309d137bf8ae5f9879"
Sep 11 21:46:21.798668 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:46:21.798648    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4986ef68b7336d2ad7630659587734ff99c00cd236ab3309d137bf8ae5f9879\": container with ID starting with a4986ef68b7336d2ad7630659587734ff99c00cd236ab3309d137bf8ae5f9879 not found: ID does not exist" containerID="a4986ef68b7336d2ad7630659587734ff99c00cd236ab3309d137bf8ae5f9879"
Sep 11 21:46:21.798747 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.798682    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4986ef68b7336d2ad7630659587734ff99c00cd236ab3309d137bf8ae5f9879"} err="failed to get container status \"a4986ef68b7336d2ad7630659587734ff99c00cd236ab3309d137bf8ae5f9879\": rpc error: code = NotFound desc = could not find container \"a4986ef68b7336d2ad7630659587734ff99c00cd236ab3309d137bf8ae5f9879\": container with ID starting with a4986ef68b7336d2ad7630659587734ff99c00cd236ab3309d137bf8ae5f9879 not found: ID does not exist"
Sep 11 21:46:21.798747 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.798705    2544 scope.go:117] "RemoveContainer" containerID="aab96f7cc0bca65569d6aa439de8be368f477bf2f0579de5764d40a5db3219cb"
Sep 11 21:46:21.806601 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.806582    2544 scope.go:117] "RemoveContainer" containerID="aab96f7cc0bca65569d6aa439de8be368f477bf2f0579de5764d40a5db3219cb"
Sep 11 21:46:21.806848 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:46:21.806826    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aab96f7cc0bca65569d6aa439de8be368f477bf2f0579de5764d40a5db3219cb\": container with ID starting with aab96f7cc0bca65569d6aa439de8be368f477bf2f0579de5764d40a5db3219cb not found: ID does not exist" containerID="aab96f7cc0bca65569d6aa439de8be368f477bf2f0579de5764d40a5db3219cb"
Sep 11 21:46:21.806901 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.806856    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aab96f7cc0bca65569d6aa439de8be368f477bf2f0579de5764d40a5db3219cb"} err="failed to get container status \"aab96f7cc0bca65569d6aa439de8be368f477bf2f0579de5764d40a5db3219cb\": rpc error: code = NotFound desc = could not find container \"aab96f7cc0bca65569d6aa439de8be368f477bf2f0579de5764d40a5db3219cb\": container with ID starting with aab96f7cc0bca65569d6aa439de8be368f477bf2f0579de5764d40a5db3219cb not found: ID does not exist"
Sep 11 21:46:21.894743 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.894656    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbt9r\" (UniqueName: \"kubernetes.io/projected/f958c5bf-e52f-4857-902e-e82172d7e4ac-kube-api-access-kbt9r\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:46:21.894743 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:21.894686    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjg7l\" (UniqueName: \"kubernetes.io/projected/a94bc216-b951-4421-af51-4a01a48389e1-kube-api-access-qjg7l\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:46:22.109403 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:22.109376    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj"]
Sep 11 21:46:22.112947 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:22.112923    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7a9aa317-56645fc5db-mbvjj"]
Sep 11 21:46:22.122887 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:22.122859    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw"]
Sep 11 21:46:22.131322 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:22.131296    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7a9aa317-55ff94b669-qftmw"]
Sep 11 21:46:23.592565 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:23.592534    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a94bc216-b951-4421-af51-4a01a48389e1" path="/var/lib/kubelet/pods/a94bc216-b951-4421-af51-4a01a48389e1/volumes"
Sep 11 21:46:23.592931 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:23.592906    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f958c5bf-e52f-4857-902e-e82172d7e4ac" path="/var/lib/kubelet/pods/f958c5bf-e52f-4857-902e-e82172d7e4ac/volumes"
Sep 11 21:46:32.431442 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.431385    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z" podUID="0e9a0319-3733-43f4-9a57-f8497035fd1a" containerName="main" containerID="cri-o://52f2449c2c1770eea56b0a6aa600ce550a5291dfa55b3ec343345fe01bc2e9ef" gracePeriod=45
Sep 11 21:46:32.680514 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.680487    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:46:32.796503 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.796474    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-tmp-dir\") pod \"0e9a0319-3733-43f4-9a57-f8497035fd1a\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") "
Sep 11 21:46:32.796672 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.796515    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-dshm\") pod \"0e9a0319-3733-43f4-9a57-f8497035fd1a\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") "
Sep 11 21:46:32.796672 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.796578    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-home\") pod \"0e9a0319-3733-43f4-9a57-f8497035fd1a\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") "
Sep 11 21:46:32.796672 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.796638    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpq8k\" (UniqueName: \"kubernetes.io/projected/0e9a0319-3733-43f4-9a57-f8497035fd1a-kube-api-access-wpq8k\") pod \"0e9a0319-3733-43f4-9a57-f8497035fd1a\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") "
Sep 11 21:46:32.796672 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.796659    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e9a0319-3733-43f4-9a57-f8497035fd1a-tls-certs\") pod \"0e9a0319-3733-43f4-9a57-f8497035fd1a\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") "
Sep 11 21:46:32.796889 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.796681    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-model-cache\") pod \"0e9a0319-3733-43f4-9a57-f8497035fd1a\" (UID: \"0e9a0319-3733-43f4-9a57-f8497035fd1a\") "
Sep 11 21:46:32.796889 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.796695    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e9a0319-3733-43f4-9a57-f8497035fd1a" (UID: "0e9a0319-3733-43f4-9a57-f8497035fd1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:46:32.796889 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.796825    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-home" (OuterVolumeSpecName: "home") pod "0e9a0319-3733-43f4-9a57-f8497035fd1a" (UID: "0e9a0319-3733-43f4-9a57-f8497035fd1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:46:32.797048 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.796966    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e9a0319-3733-43f4-9a57-f8497035fd1a" (UID: "0e9a0319-3733-43f4-9a57-f8497035fd1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:46:32.797048 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.796977    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:46:32.797048 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.797021    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:46:32.798777 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.798756    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9a0319-3733-43f4-9a57-f8497035fd1a-kube-api-access-wpq8k" (OuterVolumeSpecName: "kube-api-access-wpq8k") pod "0e9a0319-3733-43f4-9a57-f8497035fd1a" (UID: "0e9a0319-3733-43f4-9a57-f8497035fd1a"). InnerVolumeSpecName "kube-api-access-wpq8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:46:32.799023 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.799000    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e9a0319-3733-43f4-9a57-f8497035fd1a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e9a0319-3733-43f4-9a57-f8497035fd1a" (UID: "0e9a0319-3733-43f4-9a57-f8497035fd1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:46:32.799077 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.799014    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-dshm" (OuterVolumeSpecName: "dshm") pod "0e9a0319-3733-43f4-9a57-f8497035fd1a" (UID: "0e9a0319-3733-43f4-9a57-f8497035fd1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:46:32.849965 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.849936    2544 generic.go:356] "Generic (PLEG): container finished" podID="0e9a0319-3733-43f4-9a57-f8497035fd1a" containerID="52f2449c2c1770eea56b0a6aa600ce550a5291dfa55b3ec343345fe01bc2e9ef" exitCode=0
Sep 11 21:46:32.850090 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.849971    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z" event={"ID":"0e9a0319-3733-43f4-9a57-f8497035fd1a","Type":"ContainerDied","Data":"52f2449c2c1770eea56b0a6aa600ce550a5291dfa55b3ec343345fe01bc2e9ef"}
Sep 11 21:46:32.850090 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.849987    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"
Sep 11 21:46:32.850090 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.850018    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z" event={"ID":"0e9a0319-3733-43f4-9a57-f8497035fd1a","Type":"ContainerDied","Data":"b3a827a8d10850cdd86a912c7281d5c32cb9064caac53a720a6b950b3ada4efd"}
Sep 11 21:46:32.850090 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.850035    2544 scope.go:117] "RemoveContainer" containerID="52f2449c2c1770eea56b0a6aa600ce550a5291dfa55b3ec343345fe01bc2e9ef"
Sep 11 21:46:32.859360 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.859343    2544 scope.go:117] "RemoveContainer" containerID="52f2449c2c1770eea56b0a6aa600ce550a5291dfa55b3ec343345fe01bc2e9ef"
Sep 11 21:46:32.859614 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:46:32.859597    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f2449c2c1770eea56b0a6aa600ce550a5291dfa55b3ec343345fe01bc2e9ef\": container with ID starting with 52f2449c2c1770eea56b0a6aa600ce550a5291dfa55b3ec343345fe01bc2e9ef not found: ID does not exist" containerID="52f2449c2c1770eea56b0a6aa600ce550a5291dfa55b3ec343345fe01bc2e9ef"
Sep 11 21:46:32.859672 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.859622    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f2449c2c1770eea56b0a6aa600ce550a5291dfa55b3ec343345fe01bc2e9ef"} err="failed to get container status \"52f2449c2c1770eea56b0a6aa600ce550a5291dfa55b3ec343345fe01bc2e9ef\": rpc error: code = NotFound desc = could not find container \"52f2449c2c1770eea56b0a6aa600ce550a5291dfa55b3ec343345fe01bc2e9ef\": container with ID starting with 52f2449c2c1770eea56b0a6aa600ce550a5291dfa55b3ec343345fe01bc2e9ef not found: ID does not exist"
Sep 11 21:46:32.870799 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.870774    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"]
Sep 11 21:46:32.874721 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.874700    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-5b160c/autores-model-5b160c-kserve-5bfdc9dd75-xcc8z"]
Sep 11 21:46:32.898284 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.898261    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpq8k\" (UniqueName: \"kubernetes.io/projected/0e9a0319-3733-43f4-9a57-f8497035fd1a-kube-api-access-wpq8k\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:46:32.898284 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.898280    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e9a0319-3733-43f4-9a57-f8497035fd1a-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:46:32.898388 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.898290    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:46:32.898388 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:32.898299    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e9a0319-3733-43f4-9a57-f8497035fd1a-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:46:33.592141 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:33.592108    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e9a0319-3733-43f4-9a57-f8497035fd1a" path="/var/lib/kubelet/pods/0e9a0319-3733-43f4-9a57-f8497035fd1a/volumes"
Sep 11 21:46:54.569756 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:54.569720    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"]
Sep 11 21:46:54.570307 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:54.570002    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587" podUID="33a92534-26f9-425d-b6e5-756ba5c2d69f" containerName="istio-proxy" containerID="cri-o://a01f8a8e5f8d61626c2aa2b21a080871bf7228b99433a89850319fb8e773ff7a" gracePeriod=30
Sep 11 21:46:59.825794 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.825768    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:46:59.961585 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.961483    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-envoy\") pod \"33a92534-26f9-425d-b6e5-756ba5c2d69f\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") "
Sep 11 21:46:59.961585 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.961532    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-data\") pod \"33a92534-26f9-425d-b6e5-756ba5c2d69f\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") "
Sep 11 21:46:59.961585 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.961564    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a92534-26f9-425d-b6e5-756ba5c2d69f-istiod-ca-cert\") pod \"33a92534-26f9-425d-b6e5-756ba5c2d69f\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") "
Sep 11 21:46:59.961585 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.961581    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-workload-socket\") pod \"33a92534-26f9-425d-b6e5-756ba5c2d69f\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") "
Sep 11 21:46:59.961929 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.961606    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpm7l\" (UniqueName: \"kubernetes.io/projected/33a92534-26f9-425d-b6e5-756ba5c2d69f-kube-api-access-hpm7l\") pod \"33a92534-26f9-425d-b6e5-756ba5c2d69f\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") "
Sep 11 21:46:59.961929 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.961648    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-podinfo\") pod \"33a92534-26f9-425d-b6e5-756ba5c2d69f\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") "
Sep 11 21:46:59.961929 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.961729    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-credential-socket\") pod \"33a92534-26f9-425d-b6e5-756ba5c2d69f\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") "
Sep 11 21:46:59.961929 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.961755    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-workload-certs\") pod \"33a92534-26f9-425d-b6e5-756ba5c2d69f\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") "
Sep 11 21:46:59.961929 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.961802    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-token\") pod \"33a92534-26f9-425d-b6e5-756ba5c2d69f\" (UID: \"33a92534-26f9-425d-b6e5-756ba5c2d69f\") "
Sep 11 21:46:59.961929 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.961889    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "33a92534-26f9-425d-b6e5-756ba5c2d69f" (UID: "33a92534-26f9-425d-b6e5-756ba5c2d69f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:46:59.962264 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.962017    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-data" (OuterVolumeSpecName: "istio-data") pod "33a92534-26f9-425d-b6e5-756ba5c2d69f" (UID: "33a92534-26f9-425d-b6e5-756ba5c2d69f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:46:59.962264 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.962094    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a92534-26f9-425d-b6e5-756ba5c2d69f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "33a92534-26f9-425d-b6e5-756ba5c2d69f" (UID: "33a92534-26f9-425d-b6e5-756ba5c2d69f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:46:59.962374 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.962262    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "33a92534-26f9-425d-b6e5-756ba5c2d69f" (UID: "33a92534-26f9-425d-b6e5-756ba5c2d69f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:46:59.962374 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.962302    2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-data\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:46:59.962374 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.962322    2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a92534-26f9-425d-b6e5-756ba5c2d69f-istiod-ca-cert\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:46:59.962374 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.962335    2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-workload-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:46:59.962374 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.962334    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "33a92534-26f9-425d-b6e5-756ba5c2d69f" (UID: "33a92534-26f9-425d-b6e5-756ba5c2d69f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:46:59.963708 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.963682    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "33a92534-26f9-425d-b6e5-756ba5c2d69f" (UID: "33a92534-26f9-425d-b6e5-756ba5c2d69f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:46:59.963946 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.963924    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a92534-26f9-425d-b6e5-756ba5c2d69f-kube-api-access-hpm7l" (OuterVolumeSpecName: "kube-api-access-hpm7l") pod "33a92534-26f9-425d-b6e5-756ba5c2d69f" (UID: "33a92534-26f9-425d-b6e5-756ba5c2d69f"). InnerVolumeSpecName "kube-api-access-hpm7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:46:59.964035 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.964013    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-token" (OuterVolumeSpecName: "istio-token") pod "33a92534-26f9-425d-b6e5-756ba5c2d69f" (UID: "33a92534-26f9-425d-b6e5-756ba5c2d69f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:46:59.964076 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.964024    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "33a92534-26f9-425d-b6e5-756ba5c2d69f" (UID: "33a92534-26f9-425d-b6e5-756ba5c2d69f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:46:59.987010 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.986982    2544 generic.go:356] "Generic (PLEG): container finished" podID="33a92534-26f9-425d-b6e5-756ba5c2d69f" containerID="a01f8a8e5f8d61626c2aa2b21a080871bf7228b99433a89850319fb8e773ff7a" exitCode=0
Sep 11 21:46:59.987114 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.987023    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"
Sep 11 21:46:59.987114 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.987062    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587" event={"ID":"33a92534-26f9-425d-b6e5-756ba5c2d69f","Type":"ContainerDied","Data":"a01f8a8e5f8d61626c2aa2b21a080871bf7228b99433a89850319fb8e773ff7a"}
Sep 11 21:46:59.987196 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.987118    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587" event={"ID":"33a92534-26f9-425d-b6e5-756ba5c2d69f","Type":"ContainerDied","Data":"0acaba823ce3d7d2425abbc6861f0bdc867f407fa0c6fdbc43d121fd13f4a328"}
Sep 11 21:46:59.987196 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.987141    2544 scope.go:117] "RemoveContainer" containerID="a01f8a8e5f8d61626c2aa2b21a080871bf7228b99433a89850319fb8e773ff7a"
Sep 11 21:46:59.996488 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:46:59.996471    2544 scope.go:117] "RemoveContainer" containerID="b0c0843834c12de2c5b195d9b557e4e28d4bba63b27fdc76e1872553c6df9fff"
Sep 11 21:47:00.004290 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:00.004271    2544 scope.go:117] "RemoveContainer" containerID="a01f8a8e5f8d61626c2aa2b21a080871bf7228b99433a89850319fb8e773ff7a"
Sep 11 21:47:00.004541 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:47:00.004522    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01f8a8e5f8d61626c2aa2b21a080871bf7228b99433a89850319fb8e773ff7a\": container with ID starting with a01f8a8e5f8d61626c2aa2b21a080871bf7228b99433a89850319fb8e773ff7a not found: ID does not exist" containerID="a01f8a8e5f8d61626c2aa2b21a080871bf7228b99433a89850319fb8e773ff7a"
Sep 11 21:47:00.004604 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:00.004548    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01f8a8e5f8d61626c2aa2b21a080871bf7228b99433a89850319fb8e773ff7a"} err="failed to get container status \"a01f8a8e5f8d61626c2aa2b21a080871bf7228b99433a89850319fb8e773ff7a\": rpc error: code = NotFound desc = could not find container \"a01f8a8e5f8d61626c2aa2b21a080871bf7228b99433a89850319fb8e773ff7a\": container with ID starting with a01f8a8e5f8d61626c2aa2b21a080871bf7228b99433a89850319fb8e773ff7a not found: ID does not exist"
Sep 11 21:47:00.004604 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:00.004565    2544 scope.go:117] "RemoveContainer" containerID="b0c0843834c12de2c5b195d9b557e4e28d4bba63b27fdc76e1872553c6df9fff"
Sep 11 21:47:00.004792 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:47:00.004776    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c0843834c12de2c5b195d9b557e4e28d4bba63b27fdc76e1872553c6df9fff\": container with ID starting with b0c0843834c12de2c5b195d9b557e4e28d4bba63b27fdc76e1872553c6df9fff not found: ID does not exist" containerID="b0c0843834c12de2c5b195d9b557e4e28d4bba63b27fdc76e1872553c6df9fff"
Sep 11 21:47:00.004834 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:00.004797    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c0843834c12de2c5b195d9b557e4e28d4bba63b27fdc76e1872553c6df9fff"} err="failed to get container status \"b0c0843834c12de2c5b195d9b557e4e28d4bba63b27fdc76e1872553c6df9fff\": rpc error: code = NotFound desc = could not find container \"b0c0843834c12de2c5b195d9b557e4e28d4bba63b27fdc76e1872553c6df9fff\": container with ID starting with b0c0843834c12de2c5b195d9b557e4e28d4bba63b27fdc76e1872553c6df9fff not found: ID does not exist"
Sep 11 21:47:00.009466 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:00.009444    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"]
Sep 11 21:47:00.015434 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:00.015412    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-5b160c-openshift-default-6788979f8d-tv587"]
Sep 11 21:47:00.063343 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:00.063299    2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-token\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:47:00.063343 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:00.063337    2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-envoy\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:47:00.063343 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:00.063351    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpm7l\" (UniqueName: \"kubernetes.io/projected/33a92534-26f9-425d-b6e5-756ba5c2d69f-kube-api-access-hpm7l\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:47:00.063531 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:00.063363    2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a92534-26f9-425d-b6e5-756ba5c2d69f-istio-podinfo\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:47:00.063531 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:00.063376    2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-credential-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:47:00.063531 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:00.063387    2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a92534-26f9-425d-b6e5-756ba5c2d69f-workload-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:47:01.591999 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:01.591969    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33a92534-26f9-425d-b6e5-756ba5c2d69f" path="/var/lib/kubelet/pods/33a92534-26f9-425d-b6e5-756ba5c2d69f/volumes"
Sep 11 21:47:02.662169 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662132    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984"]
Sep 11 21:47:02.662580 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662505    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7d4f7fc-5d9b-402f-8959-6a027a4ed533" containerName="revoke-keys"
Sep 11 21:47:02.662580 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662517    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7d4f7fc-5d9b-402f-8959-6a027a4ed533" containerName="revoke-keys"
Sep 11 21:47:02.662580 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662527    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a94bc216-b951-4421-af51-4a01a48389e1" containerName="payload-pre-processing"
Sep 11 21:47:02.662580 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662533    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="a94bc216-b951-4421-af51-4a01a48389e1" containerName="payload-pre-processing"
Sep 11 21:47:02.662580 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662544    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f958c5bf-e52f-4857-902e-e82172d7e4ac" containerName="payload-processing"
Sep 11 21:47:02.662580 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662550    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="f958c5bf-e52f-4857-902e-e82172d7e4ac" containerName="payload-processing"
Sep 11 21:47:02.662580 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662562    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33a92534-26f9-425d-b6e5-756ba5c2d69f" containerName="istio-proxy"
Sep 11 21:47:02.662580 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662579    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="33a92534-26f9-425d-b6e5-756ba5c2d69f" containerName="istio-proxy"
Sep 11 21:47:02.662826 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662589    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33a92534-26f9-425d-b6e5-756ba5c2d69f" containerName="istio-proxy"
Sep 11 21:47:02.662826 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662594    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="33a92534-26f9-425d-b6e5-756ba5c2d69f" containerName="istio-proxy"
Sep 11 21:47:02.662826 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662603    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e9a0319-3733-43f4-9a57-f8497035fd1a" containerName="main"
Sep 11 21:47:02.662826 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662608    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e9a0319-3733-43f4-9a57-f8497035fd1a" containerName="main"
Sep 11 21:47:02.662826 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662661    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7d4f7fc-5d9b-402f-8959-6a027a4ed533" containerName="revoke-keys"
Sep 11 21:47:02.662826 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662668    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="33a92534-26f9-425d-b6e5-756ba5c2d69f" containerName="istio-proxy"
Sep 11 21:47:02.662826 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662673    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="33a92534-26f9-425d-b6e5-756ba5c2d69f" containerName="istio-proxy"
Sep 11 21:47:02.662826 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662678    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e9a0319-3733-43f4-9a57-f8497035fd1a" containerName="main"
Sep 11 21:47:02.662826 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662685    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="f958c5bf-e52f-4857-902e-e82172d7e4ac" containerName="payload-processing"
Sep 11 21:47:02.662826 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.662689    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="a94bc216-b951-4421-af51-4a01a48389e1" containerName="payload-pre-processing"
Sep 11 21:47:02.666992 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.666973    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984"
Sep 11 21:47:02.670989 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.670967    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-b75330\""
Sep 11 21:47:02.671100 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.671082    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-b75330-dockercfg-nlp7t\""
Sep 11 21:47:02.677667 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.677645    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984"]
Sep 11 21:47:02.787370 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.787333    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f307e47c-5267-42d8-8d45-fac98605a8db-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984\" (UID: \"f307e47c-5267-42d8-8d45-fac98605a8db\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984"
Sep 11 21:47:02.787535 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.787396    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnsvf\" (UniqueName: \"kubernetes.io/projected/f307e47c-5267-42d8-8d45-fac98605a8db-kube-api-access-cnsvf\") pod \"payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984\" (UID: \"f307e47c-5267-42d8-8d45-fac98605a8db\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984"
Sep 11 21:47:02.888871 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.888814    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnsvf\" (UniqueName: \"kubernetes.io/projected/f307e47c-5267-42d8-8d45-fac98605a8db-kube-api-access-cnsvf\") pod \"payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984\" (UID: \"f307e47c-5267-42d8-8d45-fac98605a8db\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984"
Sep 11 21:47:02.889058 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.888928    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f307e47c-5267-42d8-8d45-fac98605a8db-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984\" (UID: \"f307e47c-5267-42d8-8d45-fac98605a8db\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984"
Sep 11 21:47:02.889662 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.889642    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f307e47c-5267-42d8-8d45-fac98605a8db-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984\" (UID: \"f307e47c-5267-42d8-8d45-fac98605a8db\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984"
Sep 11 21:47:02.896688 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.896657    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnsvf\" (UniqueName: \"kubernetes.io/projected/f307e47c-5267-42d8-8d45-fac98605a8db-kube-api-access-cnsvf\") pod \"payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984\" (UID: \"f307e47c-5267-42d8-8d45-fac98605a8db\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984"
Sep 11 21:47:02.977628 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:02.977598    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984"
Sep 11 21:47:03.249499 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:03.249470    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984"]
Sep 11 21:47:03.251185 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:47:03.251146    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf307e47c_5267_42d8_8d45_fac98605a8db.slice/crio-f9421d9761673c6430d00ffe99bde904d2683d35479ce5e13528a20350d91aa8 WatchSource:0}: Error finding container f9421d9761673c6430d00ffe99bde904d2683d35479ce5e13528a20350d91aa8: Status 404 returned error can't find the container with id f9421d9761673c6430d00ffe99bde904d2683d35479ce5e13528a20350d91aa8
Sep 11 21:47:04.009863 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:04.009828    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984" event={"ID":"f307e47c-5267-42d8-8d45-fac98605a8db","Type":"ContainerStarted","Data":"cfb21241c44d217d246cb48f21681ee29e6151da73777728b1a2445cdc531608"}
Sep 11 21:47:04.010294 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:04.009888    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984" event={"ID":"f307e47c-5267-42d8-8d45-fac98605a8db","Type":"ContainerStarted","Data":"f9421d9761673c6430d00ffe99bde904d2683d35479ce5e13528a20350d91aa8"}
Sep 11 21:47:04.010294 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:04.009916    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984"
Sep 11 21:47:04.026442 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:04.026394    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984" podStartSLOduration=2.026378782 podStartE2EDuration="2.026378782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:47:04.025262064 +0000 UTC m=+986.994678885" watchObservedRunningTime="2026-09-11 21:47:04.026378782 +0000 UTC m=+986.995795654"
Sep 11 21:47:12.961738 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:12.961702    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"]
Sep 11 21:47:12.965072 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:12.965047    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:12.967774 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:12.967745    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-35e777\"/\"premium-w4-35e777-kserve-self-signed-certs\""
Sep 11 21:47:12.975792 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:12.975767    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"]
Sep 11 21:47:13.082046 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.082002    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ea762c-f334-4f8e-baa2-03a183358369-tls-certs\") pod \"premium-w4-35e777-kserve-b8cfc7758-qtcn2\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") " pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.082046 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.082051    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-model-cache\") pod \"premium-w4-35e777-kserve-b8cfc7758-qtcn2\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") " pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.082367 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.082070    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-home\") pod \"premium-w4-35e777-kserve-b8cfc7758-qtcn2\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") " pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.082367 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.082122    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-tmp-dir\") pod \"premium-w4-35e777-kserve-b8cfc7758-qtcn2\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") " pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.082367 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.082175    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-dshm\") pod \"premium-w4-35e777-kserve-b8cfc7758-qtcn2\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") " pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.082367 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.082296    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8h5v\" (UniqueName: \"kubernetes.io/projected/f8ea762c-f334-4f8e-baa2-03a183358369-kube-api-access-j8h5v\") pod \"premium-w4-35e777-kserve-b8cfc7758-qtcn2\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") " pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.183631 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.183577    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ea762c-f334-4f8e-baa2-03a183358369-tls-certs\") pod \"premium-w4-35e777-kserve-b8cfc7758-qtcn2\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") " pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.183631 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.183640    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-model-cache\") pod \"premium-w4-35e777-kserve-b8cfc7758-qtcn2\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") " pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.183896 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.183656    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-home\") pod \"premium-w4-35e777-kserve-b8cfc7758-qtcn2\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") " pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.183896 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.183672    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-tmp-dir\") pod \"premium-w4-35e777-kserve-b8cfc7758-qtcn2\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") " pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.183896 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.183693    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-dshm\") pod \"premium-w4-35e777-kserve-b8cfc7758-qtcn2\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") " pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.183896 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.183733    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8h5v\" (UniqueName: \"kubernetes.io/projected/f8ea762c-f334-4f8e-baa2-03a183358369-kube-api-access-j8h5v\") pod \"premium-w4-35e777-kserve-b8cfc7758-qtcn2\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") " pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.184113 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.184048    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-home\") pod \"premium-w4-35e777-kserve-b8cfc7758-qtcn2\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") " pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.184113 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.184092    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-model-cache\") pod \"premium-w4-35e777-kserve-b8cfc7758-qtcn2\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") " pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.184206 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.184163    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-tmp-dir\") pod \"premium-w4-35e777-kserve-b8cfc7758-qtcn2\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") " pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.185919 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.185896    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-dshm\") pod \"premium-w4-35e777-kserve-b8cfc7758-qtcn2\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") " pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.186246 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.186211    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ea762c-f334-4f8e-baa2-03a183358369-tls-certs\") pod \"premium-w4-35e777-kserve-b8cfc7758-qtcn2\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") " pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.191660 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.191639    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8h5v\" (UniqueName: \"kubernetes.io/projected/f8ea762c-f334-4f8e-baa2-03a183358369-kube-api-access-j8h5v\") pod \"premium-w4-35e777-kserve-b8cfc7758-qtcn2\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") " pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.275048 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.274996    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:13.542855 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:13.542825    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"]
Sep 11 21:47:13.544848 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:47:13.544812    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8ea762c_f334_4f8e_baa2_03a183358369.slice/crio-60c92bccdeacfe78952b24f45e490294ebdce586e91b8e3942232057f306827c WatchSource:0}: Error finding container 60c92bccdeacfe78952b24f45e490294ebdce586e91b8e3942232057f306827c: Status 404 returned error can't find the container with id 60c92bccdeacfe78952b24f45e490294ebdce586e91b8e3942232057f306827c
Sep 11 21:47:14.059542 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:14.059504    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2" event={"ID":"f8ea762c-f334-4f8e-baa2-03a183358369","Type":"ContainerStarted","Data":"6ebd0e3ec611f81bcceac85ab8bce12b198f0708813eff886e9d93956428603e"}
Sep 11 21:47:14.059542 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:14.059544    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2" event={"ID":"f8ea762c-f334-4f8e-baa2-03a183358369","Type":"ContainerStarted","Data":"60c92bccdeacfe78952b24f45e490294ebdce586e91b8e3942232057f306827c"}
Sep 11 21:47:14.076602 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:14.076546    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2" podStartSLOduration=2.076532428 podStartE2EDuration="2.076532428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:47:14.076148561 +0000 UTC m=+997.045565379" watchObservedRunningTime="2026-09-11 21:47:14.076532428 +0000 UTC m=+997.045949246"
Sep 11 21:47:15.018416 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:15.018382    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984"
Sep 11 21:47:23.276070 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:23.276029    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:23.276584 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:23.276095    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:23.288592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:23.288564    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:24.122324 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:24.122296    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:47:30.259967 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.259933    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"]
Sep 11 21:47:30.263490 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.263474    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.265966 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.265949    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f12741\"/\"premium-w0-f12741-kserve-self-signed-certs\""
Sep 11 21:47:30.274186 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.274157    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"]
Sep 11 21:47:30.352317 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.352284    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-tmp-dir\") pod \"premium-w0-f12741-kserve-c57b5578c-gjhnm\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") " pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.352477 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.352339    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-model-cache\") pod \"premium-w0-f12741-kserve-c57b5578c-gjhnm\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") " pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.352477 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.352391    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d25379-ff07-4c27-92a4-9e1a391c6e36-tls-certs\") pod \"premium-w0-f12741-kserve-c57b5578c-gjhnm\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") " pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.352477 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.352434    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-dshm\") pod \"premium-w0-f12741-kserve-c57b5578c-gjhnm\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") " pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.352582 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.352519    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-647zp\" (UniqueName: \"kubernetes.io/projected/62d25379-ff07-4c27-92a4-9e1a391c6e36-kube-api-access-647zp\") pod \"premium-w0-f12741-kserve-c57b5578c-gjhnm\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") " pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.352582 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.352540    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-home\") pod \"premium-w0-f12741-kserve-c57b5578c-gjhnm\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") " pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.453704 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.453668    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-647zp\" (UniqueName: \"kubernetes.io/projected/62d25379-ff07-4c27-92a4-9e1a391c6e36-kube-api-access-647zp\") pod \"premium-w0-f12741-kserve-c57b5578c-gjhnm\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") " pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.453704 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.453709    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-home\") pod \"premium-w0-f12741-kserve-c57b5578c-gjhnm\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") " pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.454063 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.453754    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-tmp-dir\") pod \"premium-w0-f12741-kserve-c57b5578c-gjhnm\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") " pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.454063 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.453794    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-model-cache\") pod \"premium-w0-f12741-kserve-c57b5578c-gjhnm\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") " pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.454063 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.453817    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d25379-ff07-4c27-92a4-9e1a391c6e36-tls-certs\") pod \"premium-w0-f12741-kserve-c57b5578c-gjhnm\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") " pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.454063 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.453844    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-dshm\") pod \"premium-w0-f12741-kserve-c57b5578c-gjhnm\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") " pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.454325 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.454247    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-tmp-dir\") pod \"premium-w0-f12741-kserve-c57b5578c-gjhnm\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") " pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.454325 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.454279    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-home\") pod \"premium-w0-f12741-kserve-c57b5578c-gjhnm\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") " pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.454325 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.454256    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-model-cache\") pod \"premium-w0-f12741-kserve-c57b5578c-gjhnm\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") " pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.456062 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.456046    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-dshm\") pod \"premium-w0-f12741-kserve-c57b5578c-gjhnm\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") " pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.456192 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.456173    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d25379-ff07-4c27-92a4-9e1a391c6e36-tls-certs\") pod \"premium-w0-f12741-kserve-c57b5578c-gjhnm\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") " pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.463029 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.463009    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-647zp\" (UniqueName: \"kubernetes.io/projected/62d25379-ff07-4c27-92a4-9e1a391c6e36-kube-api-access-647zp\") pod \"premium-w0-f12741-kserve-c57b5578c-gjhnm\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") " pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.572580 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.572476    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:30.841189 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:30.841163    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"]
Sep 11 21:47:30.843313 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:47:30.843283    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62d25379_ff07_4c27_92a4_9e1a391c6e36.slice/crio-f15c5f1bdac1a141dec004b4944d837093947dfa55afa3742e92323985bb9687 WatchSource:0}: Error finding container f15c5f1bdac1a141dec004b4944d837093947dfa55afa3742e92323985bb9687: Status 404 returned error can't find the container with id f15c5f1bdac1a141dec004b4944d837093947dfa55afa3742e92323985bb9687
Sep 11 21:47:31.145817 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:31.145731    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm" event={"ID":"62d25379-ff07-4c27-92a4-9e1a391c6e36","Type":"ContainerStarted","Data":"e571aadad0e61ca42f9e6d1292aaf8484ef87e27b8d6c82fcb4d47d454ddbae0"}
Sep 11 21:47:31.145817 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:31.145773    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm" event={"ID":"62d25379-ff07-4c27-92a4-9e1a391c6e36","Type":"ContainerStarted","Data":"f15c5f1bdac1a141dec004b4944d837093947dfa55afa3742e92323985bb9687"}
Sep 11 21:47:31.162622 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:31.162573    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm" podStartSLOduration=1.162558582 podStartE2EDuration="1.162558582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:47:31.161172739 +0000 UTC m=+1014.130589553" watchObservedRunningTime="2026-09-11 21:47:31.162558582 +0000 UTC m=+1014.131975400"
Sep 11 21:47:40.572883 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:40.572839    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:40.573284 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:40.572900    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:40.584691 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:40.584664    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:41.207135 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:41.207103    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:47:42.060590 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.060554    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"]
Sep 11 21:47:42.065395 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.065369    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.068794 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.068768    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7e6f5e\"/\"kube-root-ca.crt\""
Sep 11 21:47:42.068995 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.068981    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7e6f5e\"/\"openshift-service-ca.crt\""
Sep 11 21:47:42.069114 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.069087    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7e6f5e\"/\"test-model-7e6f5e-kserve-self-signed-certs\""
Sep 11 21:47:42.069195 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.069173    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7e6f5e\"/\"default-dockercfg-h7ffs\""
Sep 11 21:47:42.072420 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.072394    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"]
Sep 11 21:47:42.164407 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.164374    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-home\") pod \"test-model-7e6f5e-kserve-67754f8fc-vwckj\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") " pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.164407 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.164411    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-dshm\") pod \"test-model-7e6f5e-kserve-67754f8fc-vwckj\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") " pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.164607 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.164451    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-model-cache\") pod \"test-model-7e6f5e-kserve-67754f8fc-vwckj\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") " pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.164607 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.164503    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53b2b57-89dd-4687-ad47-4cddbcbea855-tls-certs\") pod \"test-model-7e6f5e-kserve-67754f8fc-vwckj\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") " pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.164607 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.164524    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmx9v\" (UniqueName: \"kubernetes.io/projected/f53b2b57-89dd-4687-ad47-4cddbcbea855-kube-api-access-nmx9v\") pod \"test-model-7e6f5e-kserve-67754f8fc-vwckj\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") " pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.164607 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.164540    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-tmp-dir\") pod \"test-model-7e6f5e-kserve-67754f8fc-vwckj\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") " pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.265468 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.265431    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-tmp-dir\") pod \"test-model-7e6f5e-kserve-67754f8fc-vwckj\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") " pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.265684 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.265552    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-home\") pod \"test-model-7e6f5e-kserve-67754f8fc-vwckj\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") " pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.265684 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.265589    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-dshm\") pod \"test-model-7e6f5e-kserve-67754f8fc-vwckj\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") " pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.265684 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.265666    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-model-cache\") pod \"test-model-7e6f5e-kserve-67754f8fc-vwckj\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") " pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.265883 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.265732    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53b2b57-89dd-4687-ad47-4cddbcbea855-tls-certs\") pod \"test-model-7e6f5e-kserve-67754f8fc-vwckj\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") " pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.265883 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.265767    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmx9v\" (UniqueName: \"kubernetes.io/projected/f53b2b57-89dd-4687-ad47-4cddbcbea855-kube-api-access-nmx9v\") pod \"test-model-7e6f5e-kserve-67754f8fc-vwckj\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") " pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.266567 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.266543    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-tmp-dir\") pod \"test-model-7e6f5e-kserve-67754f8fc-vwckj\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") " pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.267056 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.266901    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-home\") pod \"test-model-7e6f5e-kserve-67754f8fc-vwckj\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") " pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.267056 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.266998    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-model-cache\") pod \"test-model-7e6f5e-kserve-67754f8fc-vwckj\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") " pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.268793 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.268673    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-dshm\") pod \"test-model-7e6f5e-kserve-67754f8fc-vwckj\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") " pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.268793 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.268724    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53b2b57-89dd-4687-ad47-4cddbcbea855-tls-certs\") pod \"test-model-7e6f5e-kserve-67754f8fc-vwckj\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") " pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.274889 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.274860    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmx9v\" (UniqueName: \"kubernetes.io/projected/f53b2b57-89dd-4687-ad47-4cddbcbea855-kube-api-access-nmx9v\") pod \"test-model-7e6f5e-kserve-67754f8fc-vwckj\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") " pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.377081 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.376982    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:42.648053 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:42.648025    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"]
Sep 11 21:47:42.649272 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:47:42.649249    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf53b2b57_89dd_4687_ad47_4cddbcbea855.slice/crio-4da90a875034117bc68c6adde6526e310441a3052eed6a2d55b2a7d5805fcf3c WatchSource:0}: Error finding container 4da90a875034117bc68c6adde6526e310441a3052eed6a2d55b2a7d5805fcf3c: Status 404 returned error can't find the container with id 4da90a875034117bc68c6adde6526e310441a3052eed6a2d55b2a7d5805fcf3c
Sep 11 21:47:43.207591 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:43.207552    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj" event={"ID":"f53b2b57-89dd-4687-ad47-4cddbcbea855","Type":"ContainerStarted","Data":"bda028e993fca5b4dcf69f45c906078b05bbc0f521b898c81d4bb9f8a0743112"}
Sep 11 21:47:43.207991 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:43.207601    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj" event={"ID":"f53b2b57-89dd-4687-ad47-4cddbcbea855","Type":"ContainerStarted","Data":"4da90a875034117bc68c6adde6526e310441a3052eed6a2d55b2a7d5805fcf3c"}
Sep 11 21:47:43.229517 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:43.228033    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj" podStartSLOduration=1.228014056 podStartE2EDuration="1.228014056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:47:43.225011438 +0000 UTC m=+1026.194428248" watchObservedRunningTime="2026-09-11 21:47:43.228014056 +0000 UTC m=+1026.197430877"
Sep 11 21:47:52.377995 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:52.377957    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:52.378406 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:52.378110    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:52.390791 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:52.390763    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:47:53.269473 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:47:53.269442    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:48:10.375666 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:10.375632    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k"]
Sep 11 21:48:10.378898 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:10.378881    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k"
Sep 11 21:48:10.382067 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:10.382043    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a21586a5-dockercfg-rrm4z\""
Sep 11 21:48:10.382529 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:10.382510    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a21586a5\""
Sep 11 21:48:10.390511 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:10.390493    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k"]
Sep 11 21:48:10.446286 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:10.446255    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k\" (UID: \"3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k"
Sep 11 21:48:10.446471 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:10.446361    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd6ml\" (UniqueName: \"kubernetes.io/projected/3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd-kube-api-access-fd6ml\") pod \"payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k\" (UID: \"3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k"
Sep 11 21:48:10.547183 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:10.547140    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k\" (UID: \"3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k"
Sep 11 21:48:10.547405 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:10.547260    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd6ml\" (UniqueName: \"kubernetes.io/projected/3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd-kube-api-access-fd6ml\") pod \"payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k\" (UID: \"3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k"
Sep 11 21:48:10.547887 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:10.547868    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k\" (UID: \"3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k"
Sep 11 21:48:10.556034 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:10.556002    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd6ml\" (UniqueName: \"kubernetes.io/projected/3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd-kube-api-access-fd6ml\") pod \"payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k\" (UID: \"3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k"
Sep 11 21:48:10.689186 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:10.689103    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k"
Sep 11 21:48:10.958685 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:10.958662    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k"]
Sep 11 21:48:10.960250 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:48:10.960209    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3aba9d_8555_4e9f_92e6_b82ba3ce55dd.slice/crio-f00f96374a6df3a4c8e47cbb6f08b3611612d0dd9eaad4fe49f9691175997b12 WatchSource:0}: Error finding container f00f96374a6df3a4c8e47cbb6f08b3611612d0dd9eaad4fe49f9691175997b12: Status 404 returned error can't find the container with id f00f96374a6df3a4c8e47cbb6f08b3611612d0dd9eaad4fe49f9691175997b12
Sep 11 21:48:11.353370 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:11.353327    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k" event={"ID":"3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd","Type":"ContainerStarted","Data":"eae6cbcdc4260b79315c772ee7673745977ab72fb7d005f845845c5f49333eba"}
Sep 11 21:48:11.353559 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:11.353393    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k" event={"ID":"3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd","Type":"ContainerStarted","Data":"f00f96374a6df3a4c8e47cbb6f08b3611612d0dd9eaad4fe49f9691175997b12"}
Sep 11 21:48:11.353559 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:11.353516    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k"
Sep 11 21:48:11.369330 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:11.369267    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k" podStartSLOduration=1.36925007 podStartE2EDuration="1.36925007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:48:11.368947145 +0000 UTC m=+1054.338363962" watchObservedRunningTime="2026-09-11 21:48:11.36925007 +0000 UTC m=+1054.338666887"
Sep 11 21:48:22.363074 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:22.363045    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k"
Sep 11 21:48:25.425141 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:25.425108    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b75330-xg5cn"]
Sep 11 21:48:25.428790 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:25.428774    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b75330-xg5cn"
Sep 11 21:48:25.431114 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:25.431091    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 21:48:25.431233 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:25.431128    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:48:25.435505 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:25.435481    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b75330-xg5cn"]
Sep 11 21:48:25.594142 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:25.594114    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa770133-6b6d-48c4-a07e-9072eccb3c2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b75330-xg5cn\" (UID: \"fa770133-6b6d-48c4-a07e-9072eccb3c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b75330-xg5cn"
Sep 11 21:48:25.695094 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:25.695011    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa770133-6b6d-48c4-a07e-9072eccb3c2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b75330-xg5cn\" (UID: \"fa770133-6b6d-48c4-a07e-9072eccb3c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b75330-xg5cn"
Sep 11 21:48:25.695666 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:25.695647    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa770133-6b6d-48c4-a07e-9072eccb3c2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b75330-xg5cn\" (UID: \"fa770133-6b6d-48c4-a07e-9072eccb3c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b75330-xg5cn"
Sep 11 21:48:25.741344 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:25.741306    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b75330-xg5cn"
Sep 11 21:48:26.024699 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:26.024532    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b75330-xg5cn"]
Sep 11 21:48:26.026971 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:48:26.026942    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa770133_6b6d_48c4_a07e_9072eccb3c2c.slice/crio-575fbe69d9e7aeb30a00211b359f54246d8c8518bf0fa4506dde63ff4a65a467 WatchSource:0}: Error finding container 575fbe69d9e7aeb30a00211b359f54246d8c8518bf0fa4506dde63ff4a65a467: Status 404 returned error can't find the container with id 575fbe69d9e7aeb30a00211b359f54246d8c8518bf0fa4506dde63ff4a65a467
Sep 11 21:48:26.431711 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:26.431676    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b75330-xg5cn" event={"ID":"fa770133-6b6d-48c4-a07e-9072eccb3c2c","Type":"ContainerStarted","Data":"f018e6f3f6b282bc00ce9e50aff16df1b70d04f59145f9d1ea3a2cccfe285e96"}
Sep 11 21:48:26.432035 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:26.431726    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b75330-xg5cn" event={"ID":"fa770133-6b6d-48c4-a07e-9072eccb3c2c","Type":"ContainerStarted","Data":"575fbe69d9e7aeb30a00211b359f54246d8c8518bf0fa4506dde63ff4a65a467"}
Sep 11 21:48:26.449295 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:26.449250    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b75330-xg5cn" podStartSLOduration=1.4492311390000001 podStartE2EDuration="1.449231139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:48:26.446956828 +0000 UTC m=+1069.416373646" watchObservedRunningTime="2026-09-11 21:48:26.449231139 +0000 UTC m=+1069.418647949"
Sep 11 21:48:27.437816 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:27.437779    2544 generic.go:356] "Generic (PLEG): container finished" podID="fa770133-6b6d-48c4-a07e-9072eccb3c2c" containerID="f018e6f3f6b282bc00ce9e50aff16df1b70d04f59145f9d1ea3a2cccfe285e96" exitCode=0
Sep 11 21:48:27.438201 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:27.437865    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b75330-xg5cn" event={"ID":"fa770133-6b6d-48c4-a07e-9072eccb3c2c","Type":"ContainerDied","Data":"f018e6f3f6b282bc00ce9e50aff16df1b70d04f59145f9d1ea3a2cccfe285e96"}
Sep 11 21:48:28.588747 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:28.588723    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b75330-xg5cn"
Sep 11 21:48:28.725566 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:28.725542    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa770133-6b6d-48c4-a07e-9072eccb3c2c-maas-api-service-ca\") pod \"fa770133-6b6d-48c4-a07e-9072eccb3c2c\" (UID: \"fa770133-6b6d-48c4-a07e-9072eccb3c2c\") "
Sep 11 21:48:28.725880 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:28.725857    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa770133-6b6d-48c4-a07e-9072eccb3c2c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa770133-6b6d-48c4-a07e-9072eccb3c2c" (UID: "fa770133-6b6d-48c4-a07e-9072eccb3c2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:48:28.827172 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:28.827145    2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa770133-6b6d-48c4-a07e-9072eccb3c2c-maas-api-service-ca\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:48:29.450916 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:29.450885    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b75330-xg5cn"
Sep 11 21:48:29.451084 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:29.450915    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b75330-xg5cn" event={"ID":"fa770133-6b6d-48c4-a07e-9072eccb3c2c","Type":"ContainerDied","Data":"575fbe69d9e7aeb30a00211b359f54246d8c8518bf0fa4506dde63ff4a65a467"}
Sep 11 21:48:29.451084 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:29.450989    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="575fbe69d9e7aeb30a00211b359f54246d8c8518bf0fa4506dde63ff4a65a467"
Sep 11 21:48:36.909930 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:36.909898    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984"]
Sep 11 21:48:36.910410 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:36.910130    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984" podUID="f307e47c-5267-42d8-8d45-fac98605a8db" containerName="payload-pre-processing" containerID="cri-o://cfb21241c44d217d246cb48f21681ee29e6151da73777728b1a2445cdc531608" gracePeriod=30
Sep 11 21:48:37.062752 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.062715    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b75330-xg5cn"]
Sep 11 21:48:37.064848 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.064821    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b75330-xg5cn"]
Sep 11 21:48:37.163054 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.162971    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984"
Sep 11 21:48:37.206551 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.206520    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnsvf\" (UniqueName: \"kubernetes.io/projected/f307e47c-5267-42d8-8d45-fac98605a8db-kube-api-access-cnsvf\") pod \"f307e47c-5267-42d8-8d45-fac98605a8db\" (UID: \"f307e47c-5267-42d8-8d45-fac98605a8db\") "
Sep 11 21:48:37.206551 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.206554    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f307e47c-5267-42d8-8d45-fac98605a8db-plugins-config-volume\") pod \"f307e47c-5267-42d8-8d45-fac98605a8db\" (UID: \"f307e47c-5267-42d8-8d45-fac98605a8db\") "
Sep 11 21:48:37.206960 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.206929    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f307e47c-5267-42d8-8d45-fac98605a8db-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f307e47c-5267-42d8-8d45-fac98605a8db" (UID: "f307e47c-5267-42d8-8d45-fac98605a8db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:48:37.208678 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.208652    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f307e47c-5267-42d8-8d45-fac98605a8db-kube-api-access-cnsvf" (OuterVolumeSpecName: "kube-api-access-cnsvf") pod "f307e47c-5267-42d8-8d45-fac98605a8db" (UID: "f307e47c-5267-42d8-8d45-fac98605a8db"). InnerVolumeSpecName "kube-api-access-cnsvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:48:37.307651 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.307611    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnsvf\" (UniqueName: \"kubernetes.io/projected/f307e47c-5267-42d8-8d45-fac98605a8db-kube-api-access-cnsvf\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:48:37.307651 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.307648    2544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f307e47c-5267-42d8-8d45-fac98605a8db-plugins-config-volume\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:48:37.493477 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.493439    2544 generic.go:356] "Generic (PLEG): container finished" podID="f307e47c-5267-42d8-8d45-fac98605a8db" containerID="cfb21241c44d217d246cb48f21681ee29e6151da73777728b1a2445cdc531608" exitCode=0
Sep 11 21:48:37.493660 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.493491    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984"
Sep 11 21:48:37.493660 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.493519    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984" event={"ID":"f307e47c-5267-42d8-8d45-fac98605a8db","Type":"ContainerDied","Data":"cfb21241c44d217d246cb48f21681ee29e6151da73777728b1a2445cdc531608"}
Sep 11 21:48:37.493660 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.493590    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984" event={"ID":"f307e47c-5267-42d8-8d45-fac98605a8db","Type":"ContainerDied","Data":"f9421d9761673c6430d00ffe99bde904d2683d35479ce5e13528a20350d91aa8"}
Sep 11 21:48:37.493829 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.493663    2544 scope.go:117] "RemoveContainer" containerID="cfb21241c44d217d246cb48f21681ee29e6151da73777728b1a2445cdc531608"
Sep 11 21:48:37.505365 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.505332    2544 scope.go:117] "RemoveContainer" containerID="cfb21241c44d217d246cb48f21681ee29e6151da73777728b1a2445cdc531608"
Sep 11 21:48:37.505929 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:48:37.505904    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb21241c44d217d246cb48f21681ee29e6151da73777728b1a2445cdc531608\": container with ID starting with cfb21241c44d217d246cb48f21681ee29e6151da73777728b1a2445cdc531608 not found: ID does not exist" containerID="cfb21241c44d217d246cb48f21681ee29e6151da73777728b1a2445cdc531608"
Sep 11 21:48:37.506027 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.505940    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb21241c44d217d246cb48f21681ee29e6151da73777728b1a2445cdc531608"} err="failed to get container status \"cfb21241c44d217d246cb48f21681ee29e6151da73777728b1a2445cdc531608\": rpc error: code = NotFound desc = could not find container \"cfb21241c44d217d246cb48f21681ee29e6151da73777728b1a2445cdc531608\": container with ID starting with cfb21241c44d217d246cb48f21681ee29e6151da73777728b1a2445cdc531608 not found: ID does not exist"
Sep 11 21:48:37.515106 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.515086    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984"]
Sep 11 21:48:37.518836 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.518814    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b75330-78ccd7ff74-l8984"]
Sep 11 21:48:37.595358 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.593912    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f307e47c-5267-42d8-8d45-fac98605a8db" path="/var/lib/kubelet/pods/f307e47c-5267-42d8-8d45-fac98605a8db/volumes"
Sep 11 21:48:37.595358 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:37.594480    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa770133-6b6d-48c4-a07e-9072eccb3c2c" path="/var/lib/kubelet/pods/fa770133-6b6d-48c4-a07e-9072eccb3c2c/volumes"
Sep 11 21:48:43.127157 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:43.127121    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a21586a5-ldl9n"]
Sep 11 21:48:43.127634 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:43.127498    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa770133-6b6d-48c4-a07e-9072eccb3c2c" containerName="revoke-keys"
Sep 11 21:48:43.127634 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:43.127512    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa770133-6b6d-48c4-a07e-9072eccb3c2c" containerName="revoke-keys"
Sep 11 21:48:43.127634 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:43.127527    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f307e47c-5267-42d8-8d45-fac98605a8db" containerName="payload-pre-processing"
Sep 11 21:48:43.127634 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:43.127533    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="f307e47c-5267-42d8-8d45-fac98605a8db" containerName="payload-pre-processing"
Sep 11 21:48:43.127634 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:43.127591    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="f307e47c-5267-42d8-8d45-fac98605a8db" containerName="payload-pre-processing"
Sep 11 21:48:43.127634 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:43.127602    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa770133-6b6d-48c4-a07e-9072eccb3c2c" containerName="revoke-keys"
Sep 11 21:48:43.132109 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:43.132092    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a21586a5-ldl9n"
Sep 11 21:48:43.135135 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:43.135112    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 21:48:43.135440 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:43.135423    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:48:43.144849 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:43.144820    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a21586a5-ldl9n"]
Sep 11 21:48:43.161537 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:43.161507    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffc2e75-6739-4826-b947-f871e982cc20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a21586a5-ldl9n\" (UID: \"9ffc2e75-6739-4826-b947-f871e982cc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a21586a5-ldl9n"
Sep 11 21:48:43.262396 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:43.262364    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffc2e75-6739-4826-b947-f871e982cc20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a21586a5-ldl9n\" (UID: \"9ffc2e75-6739-4826-b947-f871e982cc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a21586a5-ldl9n"
Sep 11 21:48:43.262935 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:43.262916    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffc2e75-6739-4826-b947-f871e982cc20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a21586a5-ldl9n\" (UID: \"9ffc2e75-6739-4826-b947-f871e982cc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a21586a5-ldl9n"
Sep 11 21:48:43.441467 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:43.441394    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a21586a5-ldl9n"
Sep 11 21:48:43.713469 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:43.713440    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a21586a5-ldl9n"]
Sep 11 21:48:43.715181 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:48:43.715150    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffc2e75_6739_4826_b947_f871e982cc20.slice/crio-eb50cb5e23d6614047cda95cd2d448dd6cac6abc4fbd47fec21ae50488fc37c3 WatchSource:0}: Error finding container eb50cb5e23d6614047cda95cd2d448dd6cac6abc4fbd47fec21ae50488fc37c3: Status 404 returned error can't find the container with id eb50cb5e23d6614047cda95cd2d448dd6cac6abc4fbd47fec21ae50488fc37c3
Sep 11 21:48:44.529932 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:44.529901    2544 generic.go:356] "Generic (PLEG): container finished" podID="9ffc2e75-6739-4826-b947-f871e982cc20" containerID="f1b0da04ab3af3ad60eb58f535c6298059eac71bcf2745eab15457d114592158" exitCode=0
Sep 11 21:48:44.530403 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:44.529966    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a21586a5-ldl9n" event={"ID":"9ffc2e75-6739-4826-b947-f871e982cc20","Type":"ContainerDied","Data":"f1b0da04ab3af3ad60eb58f535c6298059eac71bcf2745eab15457d114592158"}
Sep 11 21:48:44.530403 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:44.530002    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a21586a5-ldl9n" event={"ID":"9ffc2e75-6739-4826-b947-f871e982cc20","Type":"ContainerStarted","Data":"eb50cb5e23d6614047cda95cd2d448dd6cac6abc4fbd47fec21ae50488fc37c3"}
Sep 11 21:48:45.680082 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:45.680057    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a21586a5-ldl9n"
Sep 11 21:48:45.787273 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:45.787195    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffc2e75-6739-4826-b947-f871e982cc20-maas-api-service-ca\") pod \"9ffc2e75-6739-4826-b947-f871e982cc20\" (UID: \"9ffc2e75-6739-4826-b947-f871e982cc20\") "
Sep 11 21:48:45.787524 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:45.787503    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ffc2e75-6739-4826-b947-f871e982cc20-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9ffc2e75-6739-4826-b947-f871e982cc20" (UID: "9ffc2e75-6739-4826-b947-f871e982cc20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:48:45.888016 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:45.887983    2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffc2e75-6739-4826-b947-f871e982cc20-maas-api-service-ca\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:48:46.541769 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:46.541740    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a21586a5-ldl9n"
Sep 11 21:48:46.541952 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:46.541739    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a21586a5-ldl9n" event={"ID":"9ffc2e75-6739-4826-b947-f871e982cc20","Type":"ContainerDied","Data":"eb50cb5e23d6614047cda95cd2d448dd6cac6abc4fbd47fec21ae50488fc37c3"}
Sep 11 21:48:46.541952 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:46.541862    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb50cb5e23d6614047cda95cd2d448dd6cac6abc4fbd47fec21ae50488fc37c3"
Sep 11 21:48:52.658343 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.658307    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"]
Sep 11 21:48:52.658833 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.658817    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ffc2e75-6739-4826-b947-f871e982cc20" containerName="revoke-keys"
Sep 11 21:48:52.658880 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.658836    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ffc2e75-6739-4826-b947-f871e982cc20" containerName="revoke-keys"
Sep 11 21:48:52.658954 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.658943    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ffc2e75-6739-4826-b947-f871e982cc20" containerName="revoke-keys"
Sep 11 21:48:52.662270 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.662247    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.665022 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.664998    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-35e777\"/\"e2e-conflict-a-09c5f82a-kserve-self-signed-certs\""
Sep 11 21:48:52.673777 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.673744    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"]
Sep 11 21:48:52.755154 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.755120    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-tls-certs\") pod \"e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.755313 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.755170    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-home\") pod \"e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.755313 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.755249    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-model-cache\") pod \"e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.755313 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.755301    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdwld\" (UniqueName: \"kubernetes.io/projected/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-kube-api-access-qdwld\") pod \"e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.755441 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.755386    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-tmp-dir\") pod \"e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.755441 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.755419    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-dshm\") pod \"e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.855992 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.855956    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-home\") pod \"e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.856159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.856006    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-model-cache\") pod \"e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.856159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.856050    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdwld\" (UniqueName: \"kubernetes.io/projected/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-kube-api-access-qdwld\") pod \"e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.856159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.856085    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-tmp-dir\") pod \"e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.856159 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.856109    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-dshm\") pod \"e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.856336 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.856158    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-tls-certs\") pod \"e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.856445 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.856421    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-home\") pod \"e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.856520 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.856481    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-model-cache\") pod \"e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.856520 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.856496    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-tmp-dir\") pod \"e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.858329 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.858311    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-dshm\") pod \"e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.858596 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.858579    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-tls-certs\") pod \"e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.863640 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.863619    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdwld\" (UniqueName: \"kubernetes.io/projected/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-kube-api-access-qdwld\") pod \"e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:52.977116 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:52.977082    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:48:53.247817 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:53.247733    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"]
Sep 11 21:48:53.250708 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:48:53.250666    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe047f39_7b76_45c8_8310_7ffbaf6dd38c.slice/crio-d10078f9a2e0383f711fac845713314f462f8cfed71ea06339deff247b3f68f0 WatchSource:0}: Error finding container d10078f9a2e0383f711fac845713314f462f8cfed71ea06339deff247b3f68f0: Status 404 returned error can't find the container with id d10078f9a2e0383f711fac845713314f462f8cfed71ea06339deff247b3f68f0
Sep 11 21:48:53.581232 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:53.581112    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5" event={"ID":"fe047f39-7b76-45c8-8310-7ffbaf6dd38c","Type":"ContainerStarted","Data":"055d35fca944f5ef05bb452a6e5d43f8be3dd46991486f248d1bb51c6c963147"}
Sep 11 21:48:53.581232 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:53.581177    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5" event={"ID":"fe047f39-7b76-45c8-8310-7ffbaf6dd38c","Type":"ContainerStarted","Data":"d10078f9a2e0383f711fac845713314f462f8cfed71ea06339deff247b3f68f0"}
Sep 11 21:48:53.600752 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:53.600697    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5" podStartSLOduration=1.600676725 podStartE2EDuration="1.600676725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:48:53.600353574 +0000 UTC m=+1096.569770393" watchObservedRunningTime="2026-09-11 21:48:53.600676725 +0000 UTC m=+1096.570093544"
Sep 11 21:48:53.656242 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:53.656193    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k"]
Sep 11 21:48:53.656460 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:53.656437    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k" podUID="3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd" containerName="payload-pre-processing" containerID="cri-o://eae6cbcdc4260b79315c772ee7673745977ab72fb7d005f845845c5f49333eba" gracePeriod=30
Sep 11 21:48:53.808081 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:53.808045    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a21586a5-ldl9n"]
Sep 11 21:48:53.810773 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:53.810748    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a21586a5-ldl9n"]
Sep 11 21:48:53.940611 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:53.940586    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k"
Sep 11 21:48:54.067592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:54.067560    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fd6ml\" (UniqueName: \"kubernetes.io/projected/3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd-kube-api-access-fd6ml\") pod \"3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd\" (UID: \"3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd\") "
Sep 11 21:48:54.067745 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:54.067691    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd-plugins-config-volume\") pod \"3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd\" (UID: \"3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd\") "
Sep 11 21:48:54.068020 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:54.067999    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd" (UID: "3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:48:54.069623 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:54.069602    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd-kube-api-access-fd6ml" (OuterVolumeSpecName: "kube-api-access-fd6ml") pod "3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd" (UID: "3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd"). InnerVolumeSpecName "kube-api-access-fd6ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:48:54.168767 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:54.168707    2544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd-plugins-config-volume\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:48:54.168767 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:54.168731    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd6ml\" (UniqueName: \"kubernetes.io/projected/3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd-kube-api-access-fd6ml\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:48:54.587856 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:54.587825    2544 generic.go:356] "Generic (PLEG): container finished" podID="3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd" containerID="eae6cbcdc4260b79315c772ee7673745977ab72fb7d005f845845c5f49333eba" exitCode=0
Sep 11 21:48:54.588044 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:54.587867    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k"
Sep 11 21:48:54.588044 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:54.587902    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k" event={"ID":"3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd","Type":"ContainerDied","Data":"eae6cbcdc4260b79315c772ee7673745977ab72fb7d005f845845c5f49333eba"}
Sep 11 21:48:54.588044 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:54.587949    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k" event={"ID":"3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd","Type":"ContainerDied","Data":"f00f96374a6df3a4c8e47cbb6f08b3611612d0dd9eaad4fe49f9691175997b12"}
Sep 11 21:48:54.588044 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:54.587977    2544 scope.go:117] "RemoveContainer" containerID="eae6cbcdc4260b79315c772ee7673745977ab72fb7d005f845845c5f49333eba"
Sep 11 21:48:54.596865 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:54.596850    2544 scope.go:117] "RemoveContainer" containerID="eae6cbcdc4260b79315c772ee7673745977ab72fb7d005f845845c5f49333eba"
Sep 11 21:48:54.597091 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:48:54.597075    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eae6cbcdc4260b79315c772ee7673745977ab72fb7d005f845845c5f49333eba\": container with ID starting with eae6cbcdc4260b79315c772ee7673745977ab72fb7d005f845845c5f49333eba not found: ID does not exist" containerID="eae6cbcdc4260b79315c772ee7673745977ab72fb7d005f845845c5f49333eba"
Sep 11 21:48:54.597138 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:54.597097    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae6cbcdc4260b79315c772ee7673745977ab72fb7d005f845845c5f49333eba"} err="failed to get container status \"eae6cbcdc4260b79315c772ee7673745977ab72fb7d005f845845c5f49333eba\": rpc error: code = NotFound desc = could not find container \"eae6cbcdc4260b79315c772ee7673745977ab72fb7d005f845845c5f49333eba\": container with ID starting with eae6cbcdc4260b79315c772ee7673745977ab72fb7d005f845845c5f49333eba not found: ID does not exist"
Sep 11 21:48:54.608463 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:54.608443    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k"]
Sep 11 21:48:54.611890 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:54.611872    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a21586a5-6db597f754-h9w6k"]
Sep 11 21:48:55.592951 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:55.592916    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd" path="/var/lib/kubelet/pods/3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd/volumes"
Sep 11 21:48:55.593426 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:48:55.593368    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ffc2e75-6739-4826-b947-f871e982cc20" path="/var/lib/kubelet/pods/9ffc2e75-6739-4826-b947-f871e982cc20/volumes"
Sep 11 21:49:02.977792 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:02.977756    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:49:02.978181 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:02.977833    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:49:02.990236 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:02.990193    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:49:03.642896 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:03.642860    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:49:25.298741 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.298707    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"]
Sep 11 21:49:25.299353 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.299042    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd" containerName="payload-pre-processing"
Sep 11 21:49:25.299353 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.299055    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd" containerName="payload-pre-processing"
Sep 11 21:49:25.299353 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.299151    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a3aba9d-8555-4e9f-92e6-b82ba3ce55dd" containerName="payload-pre-processing"
Sep 11 21:49:25.302293 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.302277    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.305570 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.305542    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-35e777\"/\"e2e-conflict-b-09c5f82a-kserve-self-signed-certs\""
Sep 11 21:49:25.325304 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.325279    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"]
Sep 11 21:49:25.348402 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.348360    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-tls-certs\") pod \"e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.348540 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.348422    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-tmp-dir\") pod \"e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.348540 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.348469    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-home\") pod \"e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.348540 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.348500    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-dshm\") pod \"e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.348696 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.348544    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-model-cache\") pod \"e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.348696 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.348590    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpgwl\" (UniqueName: \"kubernetes.io/projected/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-kube-api-access-hpgwl\") pod \"e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.450001 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.449973    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-tmp-dir\") pod \"e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.450176 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.450033    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-home\") pod \"e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.450176 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.450070    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-dshm\") pod \"e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.450176 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.450115    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-model-cache\") pod \"e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.450176 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.450148    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpgwl\" (UniqueName: \"kubernetes.io/projected/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-kube-api-access-hpgwl\") pod \"e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.450401 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.450253    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-tls-certs\") pod \"e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.450465 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.450448    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-tmp-dir\") pod \"e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.450525 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.450493    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-home\") pod \"e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.450580 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.450553    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-model-cache\") pod \"e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.452288 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.452269    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-dshm\") pod \"e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.452651 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.452633    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-tls-certs\") pod \"e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.459454 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.459434    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpgwl\" (UniqueName: \"kubernetes.io/projected/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-kube-api-access-hpgwl\") pod \"e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") " pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.611200 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.611130    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:25.895565 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:25.894288    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"]
Sep 11 21:49:25.900291 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:49:25.900256    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98a7e7f6_501c_40e0_bf93_27ed97c2ff23.slice/crio-44339ea3c6e20519cfa1438e53404240b3c3fd48f393343a63d9c7f3a286a827 WatchSource:0}: Error finding container 44339ea3c6e20519cfa1438e53404240b3c3fd48f393343a63d9c7f3a286a827: Status 404 returned error can't find the container with id 44339ea3c6e20519cfa1438e53404240b3c3fd48f393343a63d9c7f3a286a827
Sep 11 21:49:26.750900 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:26.750863    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p" event={"ID":"98a7e7f6-501c-40e0-bf93-27ed97c2ff23","Type":"ContainerStarted","Data":"30f037ff19347e808a3d77b04e36c12de7d014b7fa03963eb73755c14d281a92"}
Sep 11 21:49:26.750900 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:26.750902    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p" event={"ID":"98a7e7f6-501c-40e0-bf93-27ed97c2ff23","Type":"ContainerStarted","Data":"44339ea3c6e20519cfa1438e53404240b3c3fd48f393343a63d9c7f3a286a827"}
Sep 11 21:49:26.770182 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:26.770132    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p" podStartSLOduration=1.7701167789999999 podStartE2EDuration="1.770116779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:49:26.767593274 +0000 UTC m=+1129.737010091" watchObservedRunningTime="2026-09-11 21:49:26.770116779 +0000 UTC m=+1129.739533596"
Sep 11 21:49:35.611697 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:35.611666    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:35.612098 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:35.611720    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:35.623923 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:35.623896    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:35.809107 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:35.809078    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:49:48.290989 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.290950    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"]
Sep 11 21:49:48.295898 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.295878    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.298573 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.298549    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-afd794-openshift-default-dockercfg-tr28r\""
Sep 11 21:49:48.305022 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.304996    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"]
Sep 11 21:49:48.470859 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.470821    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-istio-envoy\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.471029 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.470863    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb93698f-4a48-4b60-be78-74623c953109-istiod-ca-cert\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.471029 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.470917    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb93698f-4a48-4b60-be78-74623c953109-istio-podinfo\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.471029 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.471006    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-workload-socket\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.471164 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.471066    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-istio-data\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.471208 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.471146    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb93698f-4a48-4b60-be78-74623c953109-istio-token\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.471279 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.471251    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-workload-certs\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.471317 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.471300    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-credential-socket\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.471369 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.471324    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj25k\" (UniqueName: \"kubernetes.io/projected/fb93698f-4a48-4b60-be78-74623c953109-kube-api-access-fj25k\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.572338 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.572248    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-workload-socket\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.572338 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.572290    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-istio-data\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.572338 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.572326    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb93698f-4a48-4b60-be78-74623c953109-istio-token\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.572615 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.572349    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-workload-certs\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.572615 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.572377    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-credential-socket\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.572615 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.572503    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj25k\" (UniqueName: \"kubernetes.io/projected/fb93698f-4a48-4b60-be78-74623c953109-kube-api-access-fj25k\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.572615 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.572570    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-istio-envoy\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.572615 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.572600    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb93698f-4a48-4b60-be78-74623c953109-istiod-ca-cert\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.572839 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.572637    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb93698f-4a48-4b60-be78-74623c953109-istio-podinfo\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.572839 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.572696    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-workload-socket\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.572839 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.572739    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-workload-certs\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.572979 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.572902    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-istio-data\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.573165 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.573137    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-credential-socket\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.573284 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.573267    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb93698f-4a48-4b60-be78-74623c953109-istiod-ca-cert\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.574928 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.574899    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-istio-envoy\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.574928 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.574921    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb93698f-4a48-4b60-be78-74623c953109-istio-podinfo\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.581294 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.581270    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb93698f-4a48-4b60-be78-74623c953109-istio-token\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.581708 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.581689    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj25k\" (UniqueName: \"kubernetes.io/projected/fb93698f-4a48-4b60-be78-74623c953109-kube-api-access-fj25k\") pod \"e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") " pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.606972 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.606949    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:48.885185 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.885155    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"]
Sep 11 21:49:48.886356 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:49:48.886328    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb93698f_4a48_4b60_be78_74623c953109.slice/crio-370a96d82ac9f40cbda949581ceafc113818fea37180536678e5432796f339f4 WatchSource:0}: Error finding container 370a96d82ac9f40cbda949581ceafc113818fea37180536678e5432796f339f4: Status 404 returned error can't find the container with id 370a96d82ac9f40cbda949581ceafc113818fea37180536678e5432796f339f4
Sep 11 21:49:48.888601 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.888568    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:49:48.888732 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.888647    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:49:48.888732 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:48.888691    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:49:49.868876 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:49.868838    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8" event={"ID":"fb93698f-4a48-4b60-be78-74623c953109","Type":"ContainerStarted","Data":"34d57f3755beb1181a142760f748eeebd59288b35f07afef2bf4957d195bc008"}
Sep 11 21:49:49.869370 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:49.868899    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8" event={"ID":"fb93698f-4a48-4b60-be78-74623c953109","Type":"ContainerStarted","Data":"370a96d82ac9f40cbda949581ceafc113818fea37180536678e5432796f339f4"}
Sep 11 21:49:49.891795 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:49.891745    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8" podStartSLOduration=1.891731061 podStartE2EDuration="1.891731061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:49:49.889239015 +0000 UTC m=+1152.858655824" watchObservedRunningTime="2026-09-11 21:49:49.891731061 +0000 UTC m=+1152.861147879"
Sep 11 21:49:50.607454 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:50.607426    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:50.612509 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:50.612487    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:50.874678 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:50.874591    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:49:50.875560 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:49:50.875535    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:50:20.875649 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:20.875576    2544 patch_prober.go:28] interesting pod/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 11 21:50:20.875649 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:20.875629    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8" podUID="fb93698f-4a48-4b60-be78-74623c953109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 11 21:50:21.032511 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:21.032480    2544 generic.go:356] "Generic (PLEG): container finished" podID="fb93698f-4a48-4b60-be78-74623c953109" containerID="34d57f3755beb1181a142760f748eeebd59288b35f07afef2bf4957d195bc008" exitCode=0
Sep 11 21:50:21.032676 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:21.032551    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8" event={"ID":"fb93698f-4a48-4b60-be78-74623c953109","Type":"ContainerDied","Data":"34d57f3755beb1181a142760f748eeebd59288b35f07afef2bf4957d195bc008"}
Sep 11 21:50:21.032967 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:21.032948    2544 scope.go:117] "RemoveContainer" containerID="34d57f3755beb1181a142760f748eeebd59288b35f07afef2bf4957d195bc008"
Sep 11 21:50:21.033998 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:21.033974    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:50:21.034059 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:21.034037    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:50:21.034104 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:21.034066    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:50:22.044064 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:22.044019    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8" event={"ID":"fb93698f-4a48-4b60-be78-74623c953109","Type":"ContainerStarted","Data":"1f0601e877b3b9a18c043558fdebd620504500b7c4f4e7f23bee52e941007fcd"}
Sep 11 21:50:22.608086 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:22.608052    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:50:22.608309 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:22.608287    2544 patch_prober.go:28] interesting pod/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 11 21:50:22.608372 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:22.608353    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8" podUID="fb93698f-4a48-4b60-be78-74623c953109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 11 21:50:23.608156 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.608117    2544 patch_prober.go:28] interesting pod/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 11 21:50:23.608564 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.608183    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8" podUID="fb93698f-4a48-4b60-be78-74623c953109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 11 21:50:23.645718 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.645686    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"]
Sep 11 21:50:23.649024 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.648998    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.651383 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.651361    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d3fe19\"/\"openshift-service-ca.crt\""
Sep 11 21:50:23.652276 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.652246    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d3fe19\"/\"default-dockercfg-jbn4v\""
Sep 11 21:50:23.652527 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.652504    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d3fe19\"/\"kube-root-ca.crt\""
Sep 11 21:50:23.652645 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.652608    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d3fe19\"/\"noten-model-d3fe19-kserve-self-signed-certs\""
Sep 11 21:50:23.658618 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.658589    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"]
Sep 11 21:50:23.831048 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.831008    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bthk\" (UniqueName: \"kubernetes.io/projected/d104c923-46e2-45aa-998c-7efd8cb2664e-kube-api-access-7bthk\") pod \"noten-model-d3fe19-kserve-74465fddc9-nwx5v\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") " pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.831378 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.831358    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-home\") pod \"noten-model-d3fe19-kserve-74465fddc9-nwx5v\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") " pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.831530 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.831513    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d104c923-46e2-45aa-998c-7efd8cb2664e-tls-certs\") pod \"noten-model-d3fe19-kserve-74465fddc9-nwx5v\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") " pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.831644 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.831631    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-tmp-dir\") pod \"noten-model-d3fe19-kserve-74465fddc9-nwx5v\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") " pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.831794 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.831778    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-dshm\") pod \"noten-model-d3fe19-kserve-74465fddc9-nwx5v\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") " pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.831933 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.831919    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-model-cache\") pod \"noten-model-d3fe19-kserve-74465fddc9-nwx5v\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") " pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.933028 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.932930    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-tmp-dir\") pod \"noten-model-d3fe19-kserve-74465fddc9-nwx5v\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") " pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.933331 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.933300    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-dshm\") pod \"noten-model-d3fe19-kserve-74465fddc9-nwx5v\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") " pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.933512 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.933493    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-model-cache\") pod \"noten-model-d3fe19-kserve-74465fddc9-nwx5v\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") " pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.933722 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.933697    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bthk\" (UniqueName: \"kubernetes.io/projected/d104c923-46e2-45aa-998c-7efd8cb2664e-kube-api-access-7bthk\") pod \"noten-model-d3fe19-kserve-74465fddc9-nwx5v\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") " pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.933868 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.933853    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-home\") pod \"noten-model-d3fe19-kserve-74465fddc9-nwx5v\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") " pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.934005 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.933991    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d104c923-46e2-45aa-998c-7efd8cb2664e-tls-certs\") pod \"noten-model-d3fe19-kserve-74465fddc9-nwx5v\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") " pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.934132 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.933535    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-tmp-dir\") pod \"noten-model-d3fe19-kserve-74465fddc9-nwx5v\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") " pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.934252 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.933871    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-model-cache\") pod \"noten-model-d3fe19-kserve-74465fddc9-nwx5v\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") " pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.934444 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.934420    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-home\") pod \"noten-model-d3fe19-kserve-74465fddc9-nwx5v\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") " pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.936837 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.936810    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-dshm\") pod \"noten-model-d3fe19-kserve-74465fddc9-nwx5v\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") " pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.937471 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.937445    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d104c923-46e2-45aa-998c-7efd8cb2664e-tls-certs\") pod \"noten-model-d3fe19-kserve-74465fddc9-nwx5v\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") " pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.942448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.942394    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bthk\" (UniqueName: \"kubernetes.io/projected/d104c923-46e2-45aa-998c-7efd8cb2664e-kube-api-access-7bthk\") pod \"noten-model-d3fe19-kserve-74465fddc9-nwx5v\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") " pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:23.961500 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:23.961468    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:24.234257 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:24.234208    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"]
Sep 11 21:50:24.235749 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:50:24.235725    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd104c923_46e2_45aa_998c_7efd8cb2664e.slice/crio-26b8e9945f24b29592e3b26b96a0cc44062985c434ae7e613ccce7beba7d2962 WatchSource:0}: Error finding container 26b8e9945f24b29592e3b26b96a0cc44062985c434ae7e613ccce7beba7d2962: Status 404 returned error can't find the container with id 26b8e9945f24b29592e3b26b96a0cc44062985c434ae7e613ccce7beba7d2962
Sep 11 21:50:24.607462 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:24.607381    2544 patch_prober.go:28] interesting pod/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 11 21:50:24.607462 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:24.607434    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8" podUID="fb93698f-4a48-4b60-be78-74623c953109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 11 21:50:25.067929 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:25.067893    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v" event={"ID":"d104c923-46e2-45aa-998c-7efd8cb2664e","Type":"ContainerStarted","Data":"3633ef11f86a35e3f1c8f7f9b6c4c6b2a4cda99dbd3be4b39e12c115543fa120"}
Sep 11 21:50:25.067929 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:25.067934    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v" event={"ID":"d104c923-46e2-45aa-998c-7efd8cb2664e","Type":"ContainerStarted","Data":"26b8e9945f24b29592e3b26b96a0cc44062985c434ae7e613ccce7beba7d2962"}
Sep 11 21:50:25.084625 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:25.084579    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v" podStartSLOduration=2.084564205 podStartE2EDuration="2.084564205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:50:25.083828744 +0000 UTC m=+1188.053245562" watchObservedRunningTime="2026-09-11 21:50:25.084564205 +0000 UTC m=+1188.053981023"
Sep 11 21:50:25.607349 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:25.607312    2544 patch_prober.go:28] interesting pod/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 11 21:50:25.607522 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:25.607387    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8" podUID="fb93698f-4a48-4b60-be78-74623c953109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 11 21:50:26.607275 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:26.607239    2544 patch_prober.go:28] interesting pod/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 11 21:50:26.607654 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:26.607306    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8" podUID="fb93698f-4a48-4b60-be78-74623c953109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 11 21:50:27.611080 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:27.611049    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:50:27.611507 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:27.611260    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:50:27.611991 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:27.611976    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:50:31.434499 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:31.434461    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e919c5-hqjrs"]
Sep 11 21:50:31.436891 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:31.436875    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e919c5-hqjrs"
Sep 11 21:50:31.439367 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:31.439338    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:50:31.439478 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:31.439340    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 21:50:31.445508 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:31.445486    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e919c5-hqjrs"]
Sep 11 21:50:31.518058 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:31.518022    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/044ef7e5-8257-4fb2-9f27-e49fc254b031-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e919c5-hqjrs\" (UID: \"044ef7e5-8257-4fb2-9f27-e49fc254b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e919c5-hqjrs"
Sep 11 21:50:31.619289 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:31.619262    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/044ef7e5-8257-4fb2-9f27-e49fc254b031-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e919c5-hqjrs\" (UID: \"044ef7e5-8257-4fb2-9f27-e49fc254b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e919c5-hqjrs"
Sep 11 21:50:31.619817 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:31.619798    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/044ef7e5-8257-4fb2-9f27-e49fc254b031-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e919c5-hqjrs\" (UID: \"044ef7e5-8257-4fb2-9f27-e49fc254b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e919c5-hqjrs"
Sep 11 21:50:31.748575 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:31.748546    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e919c5-hqjrs"
Sep 11 21:50:32.016356 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:32.016328    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e919c5-hqjrs"]
Sep 11 21:50:32.018399 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:50:32.018374    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044ef7e5_8257_4fb2_9f27_e49fc254b031.slice/crio-9bc770a57f4496a4a8f09e8be328513892c4257a92e477093463f5e279f8898d WatchSource:0}: Error finding container 9bc770a57f4496a4a8f09e8be328513892c4257a92e477093463f5e279f8898d: Status 404 returned error can't find the container with id 9bc770a57f4496a4a8f09e8be328513892c4257a92e477093463f5e279f8898d
Sep 11 21:50:32.103564 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:32.103530    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e919c5-hqjrs" event={"ID":"044ef7e5-8257-4fb2-9f27-e49fc254b031","Type":"ContainerStarted","Data":"9bc770a57f4496a4a8f09e8be328513892c4257a92e477093463f5e279f8898d"}
Sep 11 21:50:33.110162 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:33.110126    2544 generic.go:356] "Generic (PLEG): container finished" podID="044ef7e5-8257-4fb2-9f27-e49fc254b031" containerID="5e38f6db06d0b00d41bd3dddff924a226012b285704e877cf9840cd0932277a8" exitCode=0
Sep 11 21:50:33.110601 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:33.110212    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e919c5-hqjrs" event={"ID":"044ef7e5-8257-4fb2-9f27-e49fc254b031","Type":"ContainerDied","Data":"5e38f6db06d0b00d41bd3dddff924a226012b285704e877cf9840cd0932277a8"}
Sep 11 21:50:33.961698 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:33.961667    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:33.961869 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:33.961724    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:33.974500 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:33.974472    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:34.127361 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:34.127333    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:50:34.271481 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:34.271456    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e919c5-hqjrs"
Sep 11 21:50:34.349735 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:34.349704    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/044ef7e5-8257-4fb2-9f27-e49fc254b031-maas-api-service-ca\") pod \"044ef7e5-8257-4fb2-9f27-e49fc254b031\" (UID: \"044ef7e5-8257-4fb2-9f27-e49fc254b031\") "
Sep 11 21:50:34.350044 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:34.350022    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044ef7e5-8257-4fb2-9f27-e49fc254b031-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "044ef7e5-8257-4fb2-9f27-e49fc254b031" (UID: "044ef7e5-8257-4fb2-9f27-e49fc254b031"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:50:34.451066 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:34.451031    2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/044ef7e5-8257-4fb2-9f27-e49fc254b031-maas-api-service-ca\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:50:35.121360 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:35.121317    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e919c5-hqjrs" event={"ID":"044ef7e5-8257-4fb2-9f27-e49fc254b031","Type":"ContainerDied","Data":"9bc770a57f4496a4a8f09e8be328513892c4257a92e477093463f5e279f8898d"}
Sep 11 21:50:35.121526 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:35.121376    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bc770a57f4496a4a8f09e8be328513892c4257a92e477093463f5e279f8898d"
Sep 11 21:50:35.121526 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:35.121340    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e919c5-hqjrs"
Sep 11 21:50:37.633409 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:37.633381    2544 scope.go:117] "RemoveContainer" containerID="a1b884f5f4561f599b4cd1a393f68c3a18f3955a9d54bce0c880f931e481377d"
Sep 11 21:50:38.651777 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.651742    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7"]
Sep 11 21:50:38.652137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.652117    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="044ef7e5-8257-4fb2-9f27-e49fc254b031" containerName="revoke-keys"
Sep 11 21:50:38.652137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.652129    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="044ef7e5-8257-4fb2-9f27-e49fc254b031" containerName="revoke-keys"
Sep 11 21:50:38.652211 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.652205    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="044ef7e5-8257-4fb2-9f27-e49fc254b031" containerName="revoke-keys"
Sep 11 21:50:38.656477 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.656459    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7"
Sep 11 21:50:38.659041 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.659018    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9f68da5e-dockercfg-cnzm5\""
Sep 11 21:50:38.659158 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.659039    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9f68da5e\""
Sep 11 21:50:38.667015 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.666992    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7"]
Sep 11 21:50:38.767963 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.767927    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc"]
Sep 11 21:50:38.771552 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.771527    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc"
Sep 11 21:50:38.781810 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.781599    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc"]
Sep 11 21:50:38.793432 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.793408    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mll4\" (UniqueName: \"kubernetes.io/projected/9ffd593a-a8ee-47c0-b262-7b68a898db62-kube-api-access-9mll4\") pod \"payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7\" (UID: \"9ffd593a-a8ee-47c0-b262-7b68a898db62\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7"
Sep 11 21:50:38.793569 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.793444    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffd593a-a8ee-47c0-b262-7b68a898db62-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7\" (UID: \"9ffd593a-a8ee-47c0-b262-7b68a898db62\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7"
Sep 11 21:50:38.894999 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.894963    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mll4\" (UniqueName: \"kubernetes.io/projected/9ffd593a-a8ee-47c0-b262-7b68a898db62-kube-api-access-9mll4\") pod \"payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7\" (UID: \"9ffd593a-a8ee-47c0-b262-7b68a898db62\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7"
Sep 11 21:50:38.895175 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.895012    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffd593a-a8ee-47c0-b262-7b68a898db62-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7\" (UID: \"9ffd593a-a8ee-47c0-b262-7b68a898db62\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7"
Sep 11 21:50:38.895175 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.895130    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f87q9\" (UniqueName: \"kubernetes.io/projected/03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62-kube-api-access-f87q9\") pod \"payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc\" (UID: \"03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62\") " pod="openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc"
Sep 11 21:50:38.895175 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.895156    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62-plugins-config-volume\") pod \"payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc\" (UID: \"03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62\") " pod="openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc"
Sep 11 21:50:38.895817 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.895797    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffd593a-a8ee-47c0-b262-7b68a898db62-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7\" (UID: \"9ffd593a-a8ee-47c0-b262-7b68a898db62\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7"
Sep 11 21:50:38.903584 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.903534    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mll4\" (UniqueName: \"kubernetes.io/projected/9ffd593a-a8ee-47c0-b262-7b68a898db62-kube-api-access-9mll4\") pod \"payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7\" (UID: \"9ffd593a-a8ee-47c0-b262-7b68a898db62\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7"
Sep 11 21:50:38.966732 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.966696    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7"
Sep 11 21:50:38.997505 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.996425    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87q9\" (UniqueName: \"kubernetes.io/projected/03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62-kube-api-access-f87q9\") pod \"payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc\" (UID: \"03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62\") " pod="openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc"
Sep 11 21:50:38.997505 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.996473    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62-plugins-config-volume\") pod \"payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc\" (UID: \"03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62\") " pod="openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc"
Sep 11 21:50:38.997766 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:38.997625    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62-plugins-config-volume\") pod \"payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc\" (UID: \"03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62\") " pod="openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc"
Sep 11 21:50:39.005753 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:39.005719    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f87q9\" (UniqueName: \"kubernetes.io/projected/03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62-kube-api-access-f87q9\") pod \"payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc\" (UID: \"03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62\") " pod="openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc"
Sep 11 21:50:39.086711 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:39.086683    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc"
Sep 11 21:50:39.242766 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:39.242742    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7"]
Sep 11 21:50:39.244064 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:50:39.244037    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffd593a_a8ee_47c0_b262_7b68a898db62.slice/crio-f2b3f84c23351df3cc98aef370b7eeb9ac0b0e4c2f86d371b533e9939bd82a8c WatchSource:0}: Error finding container f2b3f84c23351df3cc98aef370b7eeb9ac0b0e4c2f86d371b533e9939bd82a8c: Status 404 returned error can't find the container with id f2b3f84c23351df3cc98aef370b7eeb9ac0b0e4c2f86d371b533e9939bd82a8c
Sep 11 21:50:39.357435 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:39.357408    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc"]
Sep 11 21:50:39.359581 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:50:39.359552    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03df5f5a_9d46_40a1_8eb1_05c6e0ce5f62.slice/crio-8f9c51f578339c1127db84017078a7003137d9200c193d467fa928c4185e7da4 WatchSource:0}: Error finding container 8f9c51f578339c1127db84017078a7003137d9200c193d467fa928c4185e7da4: Status 404 returned error can't find the container with id 8f9c51f578339c1127db84017078a7003137d9200c193d467fa928c4185e7da4
Sep 11 21:50:40.152291 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:40.152245    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc" event={"ID":"03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62","Type":"ContainerStarted","Data":"e5670aed7318c4552e4a95bedf7468c010cd08b72c43d7dcd247cce27781a562"}
Sep 11 21:50:40.152759 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:40.152314    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc" event={"ID":"03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62","Type":"ContainerStarted","Data":"8f9c51f578339c1127db84017078a7003137d9200c193d467fa928c4185e7da4"}
Sep 11 21:50:40.152759 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:40.152450    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc"
Sep 11 21:50:40.153892 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:40.153869    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7" event={"ID":"9ffd593a-a8ee-47c0-b262-7b68a898db62","Type":"ContainerStarted","Data":"2571c772fabaf4b2baeff525a2c5d7ee49d430b0109f5e3d946e75ec26d2c9b9"}
Sep 11 21:50:40.154022 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:40.153904    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7" event={"ID":"9ffd593a-a8ee-47c0-b262-7b68a898db62","Type":"ContainerStarted","Data":"f2b3f84c23351df3cc98aef370b7eeb9ac0b0e4c2f86d371b533e9939bd82a8c"}
Sep 11 21:50:40.154022 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:40.153983    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7"
Sep 11 21:50:40.168688 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:40.168643    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc" podStartSLOduration=2.168630101 podStartE2EDuration="2.168630101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:50:40.167714045 +0000 UTC m=+1203.137130864" watchObservedRunningTime="2026-09-11 21:50:40.168630101 +0000 UTC m=+1203.138046919"
Sep 11 21:50:40.183563 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:40.183518    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7" podStartSLOduration=2.183504644 podStartE2EDuration="2.183504644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:50:40.182759337 +0000 UTC m=+1203.152176157" watchObservedRunningTime="2026-09-11 21:50:40.183504644 +0000 UTC m=+1203.152921460"
Sep 11 21:50:41.800984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:41.800946    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e919c5-hqjrs"]
Sep 11 21:50:41.803528 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:41.803504    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e919c5-hqjrs"]
Sep 11 21:50:43.592350 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:43.592319    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="044ef7e5-8257-4fb2-9f27-e49fc254b031" path="/var/lib/kubelet/pods/044ef7e5-8257-4fb2-9f27-e49fc254b031/volumes"
Sep 11 21:50:51.161383 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:51.161352    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc"
Sep 11 21:50:51.161844 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:50:51.161541    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7"
Sep 11 21:51:03.505236 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:03.505184    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"]
Sep 11 21:51:18.524148 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:18.524110    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v" podUID="d104c923-46e2-45aa-998c-7efd8cb2664e" containerName="main" containerID="cri-o://3633ef11f86a35e3f1c8f7f9b6c4c6b2a4cda99dbd3be4b39e12c115543fa120" gracePeriod=45
Sep 11 21:51:18.777290 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:18.777212    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:51:18.909606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:18.909571    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bthk\" (UniqueName: \"kubernetes.io/projected/d104c923-46e2-45aa-998c-7efd8cb2664e-kube-api-access-7bthk\") pod \"d104c923-46e2-45aa-998c-7efd8cb2664e\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") "
Sep 11 21:51:18.909772 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:18.909637    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-home\") pod \"d104c923-46e2-45aa-998c-7efd8cb2664e\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") "
Sep 11 21:51:18.909772 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:18.909666    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-model-cache\") pod \"d104c923-46e2-45aa-998c-7efd8cb2664e\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") "
Sep 11 21:51:18.909772 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:18.909708    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-tmp-dir\") pod \"d104c923-46e2-45aa-998c-7efd8cb2664e\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") "
Sep 11 21:51:18.909905 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:18.909811    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-dshm\") pod \"d104c923-46e2-45aa-998c-7efd8cb2664e\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") "
Sep 11 21:51:18.909905 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:18.909857    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d104c923-46e2-45aa-998c-7efd8cb2664e-tls-certs\") pod \"d104c923-46e2-45aa-998c-7efd8cb2664e\" (UID: \"d104c923-46e2-45aa-998c-7efd8cb2664e\") "
Sep 11 21:51:18.910012 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:18.909904    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-home" (OuterVolumeSpecName: "home") pod "d104c923-46e2-45aa-998c-7efd8cb2664e" (UID: "d104c923-46e2-45aa-998c-7efd8cb2664e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:51:18.910012 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:18.909918    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-model-cache" (OuterVolumeSpecName: "model-cache") pod "d104c923-46e2-45aa-998c-7efd8cb2664e" (UID: "d104c923-46e2-45aa-998c-7efd8cb2664e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:51:18.910120 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:18.910019    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d104c923-46e2-45aa-998c-7efd8cb2664e" (UID: "d104c923-46e2-45aa-998c-7efd8cb2664e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:51:18.910236 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:18.910206    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:18.910373 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:18.910240    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:18.910373 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:18.910256    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:18.911909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:18.911880    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-dshm" (OuterVolumeSpecName: "dshm") pod "d104c923-46e2-45aa-998c-7efd8cb2664e" (UID: "d104c923-46e2-45aa-998c-7efd8cb2664e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:51:18.912002 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:18.911946    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d104c923-46e2-45aa-998c-7efd8cb2664e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d104c923-46e2-45aa-998c-7efd8cb2664e" (UID: "d104c923-46e2-45aa-998c-7efd8cb2664e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:51:18.912002 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:18.911997    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d104c923-46e2-45aa-998c-7efd8cb2664e-kube-api-access-7bthk" (OuterVolumeSpecName: "kube-api-access-7bthk") pod "d104c923-46e2-45aa-998c-7efd8cb2664e" (UID: "d104c923-46e2-45aa-998c-7efd8cb2664e"). InnerVolumeSpecName "kube-api-access-7bthk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:51:19.011346 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:19.011309    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d104c923-46e2-45aa-998c-7efd8cb2664e-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:19.011346 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:19.011342    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d104c923-46e2-45aa-998c-7efd8cb2664e-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:19.011346 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:19.011352    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bthk\" (UniqueName: \"kubernetes.io/projected/d104c923-46e2-45aa-998c-7efd8cb2664e-kube-api-access-7bthk\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:19.375006 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:19.374971    2544 generic.go:356] "Generic (PLEG): container finished" podID="d104c923-46e2-45aa-998c-7efd8cb2664e" containerID="3633ef11f86a35e3f1c8f7f9b6c4c6b2a4cda99dbd3be4b39e12c115543fa120" exitCode=0
Sep 11 21:51:19.375188 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:19.375013    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v" event={"ID":"d104c923-46e2-45aa-998c-7efd8cb2664e","Type":"ContainerDied","Data":"3633ef11f86a35e3f1c8f7f9b6c4c6b2a4cda99dbd3be4b39e12c115543fa120"}
Sep 11 21:51:19.375188 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:19.375040    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v" event={"ID":"d104c923-46e2-45aa-998c-7efd8cb2664e","Type":"ContainerDied","Data":"26b8e9945f24b29592e3b26b96a0cc44062985c434ae7e613ccce7beba7d2962"}
Sep 11 21:51:19.375188 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:19.375055    2544 scope.go:117] "RemoveContainer" containerID="3633ef11f86a35e3f1c8f7f9b6c4c6b2a4cda99dbd3be4b39e12c115543fa120"
Sep 11 21:51:19.375188 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:19.375014    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"
Sep 11 21:51:19.384683 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:19.384662    2544 scope.go:117] "RemoveContainer" containerID="3633ef11f86a35e3f1c8f7f9b6c4c6b2a4cda99dbd3be4b39e12c115543fa120"
Sep 11 21:51:19.384938 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:51:19.384919    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3633ef11f86a35e3f1c8f7f9b6c4c6b2a4cda99dbd3be4b39e12c115543fa120\": container with ID starting with 3633ef11f86a35e3f1c8f7f9b6c4c6b2a4cda99dbd3be4b39e12c115543fa120 not found: ID does not exist" containerID="3633ef11f86a35e3f1c8f7f9b6c4c6b2a4cda99dbd3be4b39e12c115543fa120"
Sep 11 21:51:19.391752 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:19.385287    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3633ef11f86a35e3f1c8f7f9b6c4c6b2a4cda99dbd3be4b39e12c115543fa120"} err="failed to get container status \"3633ef11f86a35e3f1c8f7f9b6c4c6b2a4cda99dbd3be4b39e12c115543fa120\": rpc error: code = NotFound desc = could not find container \"3633ef11f86a35e3f1c8f7f9b6c4c6b2a4cda99dbd3be4b39e12c115543fa120\": container with ID starting with 3633ef11f86a35e3f1c8f7f9b6c4c6b2a4cda99dbd3be4b39e12c115543fa120 not found: ID does not exist"
Sep 11 21:51:19.399531 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:19.399502    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"]
Sep 11 21:51:19.402434 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:19.402407    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-d3fe19/noten-model-d3fe19-kserve-74465fddc9-nwx5v"]
Sep 11 21:51:19.592416 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:19.592385    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d104c923-46e2-45aa-998c-7efd8cb2664e" path="/var/lib/kubelet/pods/d104c923-46e2-45aa-998c-7efd8cb2664e/volumes"
Sep 11 21:51:36.166277 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:36.166174    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"]
Sep 11 21:51:37.651748 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:37.651713    2544 scope.go:117] "RemoveContainer" containerID="ed374633dc24a7ce86e8b65f5ee2fcb40bb47d2b6fc1141ae3418312773637b6"
Sep 11 21:51:37.659976 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:37.659944    2544 scope.go:117] "RemoveContainer" containerID="7e31139980c1e7d7c85e73107ef79ef311b71af4eb2e9921ead843053a4166f5"
Sep 11 21:51:39.166271 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:39.166234    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"]
Sep 11 21:51:39.166679 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:39.166444    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m" podUID="794b359e-9757-4166-aa5d-f45daae91981" containerName="istio-proxy" containerID="cri-o://6fb52e4bee67ccd22dcef7009a6b7f5065f47214c36456912c865563981aa4c7" gracePeriod=30
Sep 11 21:51:39.475485 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:39.475453    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch"]
Sep 11 21:51:39.475891 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:39.475875    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d104c923-46e2-45aa-998c-7efd8cb2664e" containerName="main"
Sep 11 21:51:39.475977 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:39.475895    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="d104c923-46e2-45aa-998c-7efd8cb2664e" containerName="main"
Sep 11 21:51:39.476032 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:39.476021    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="d104c923-46e2-45aa-998c-7efd8cb2664e" containerName="main"
Sep 11 21:51:39.480480 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:39.480459    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch"
Sep 11 21:51:39.483104 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:39.483083    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 21:51:39.483252 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:39.483134    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:51:39.489396 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:39.489365    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch"]
Sep 11 21:51:39.517212 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:39.517184    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e56b3a3-6b32-4c68-996c-f5a523d07957-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch\" (UID: \"9e56b3a3-6b32-4c68-996c-f5a523d07957\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch"
Sep 11 21:51:39.618291 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:39.618250    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e56b3a3-6b32-4c68-996c-f5a523d07957-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch\" (UID: \"9e56b3a3-6b32-4c68-996c-f5a523d07957\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch"
Sep 11 21:51:39.619138 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:39.619115    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e56b3a3-6b32-4c68-996c-f5a523d07957-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch\" (UID: \"9e56b3a3-6b32-4c68-996c-f5a523d07957\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch"
Sep 11 21:51:39.791111 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:39.791015    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch"
Sep 11 21:51:40.064563 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:40.064537    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch"]
Sep 11 21:51:40.066478 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:51:40.066452    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e56b3a3_6b32_4c68_996c_f5a523d07957.slice/crio-dfedaf49e3e91b7a987dc030d5b537a1d9287061be56038d72c8ad2f9760d7cc WatchSource:0}: Error finding container dfedaf49e3e91b7a987dc030d5b537a1d9287061be56038d72c8ad2f9760d7cc: Status 404 returned error can't find the container with id dfedaf49e3e91b7a987dc030d5b537a1d9287061be56038d72c8ad2f9760d7cc
Sep 11 21:51:40.367558 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:40.367521    2544 patch_prober.go:28] interesting pod/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 21:51:40.367941 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:40.367576    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m" podUID="794b359e-9757-4166-aa5d-f45daae91981" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 21:51:40.487943 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:40.487906    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch" event={"ID":"9e56b3a3-6b32-4c68-996c-f5a523d07957","Type":"ContainerStarted","Data":"2d68d2af820237a888f291fb52f13203324988a7113806e75ed180a6d466b6b2"}
Sep 11 21:51:40.488113 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:40.487984    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch" event={"ID":"9e56b3a3-6b32-4c68-996c-f5a523d07957","Type":"ContainerStarted","Data":"dfedaf49e3e91b7a987dc030d5b537a1d9287061be56038d72c8ad2f9760d7cc"}
Sep 11 21:51:40.505594 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:40.505549    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch" podStartSLOduration=1.5055364519999999 podStartE2EDuration="1.505536452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:51:40.503976838 +0000 UTC m=+1263.473393657" watchObservedRunningTime="2026-09-11 21:51:40.505536452 +0000 UTC m=+1263.474953317"
Sep 11 21:51:41.494635 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:41.494602    2544 generic.go:356] "Generic (PLEG): container finished" podID="9e56b3a3-6b32-4c68-996c-f5a523d07957" containerID="2d68d2af820237a888f291fb52f13203324988a7113806e75ed180a6d466b6b2" exitCode=0
Sep 11 21:51:41.495008 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:41.494653    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch" event={"ID":"9e56b3a3-6b32-4c68-996c-f5a523d07957","Type":"ContainerDied","Data":"2d68d2af820237a888f291fb52f13203324988a7113806e75ed180a6d466b6b2"}
Sep 11 21:51:42.627618 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:42.627589    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch"
Sep 11 21:51:42.749833 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:42.749798    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e56b3a3-6b32-4c68-996c-f5a523d07957-maas-api-service-ca\") pod \"9e56b3a3-6b32-4c68-996c-f5a523d07957\" (UID: \"9e56b3a3-6b32-4c68-996c-f5a523d07957\") "
Sep 11 21:51:42.750145 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:42.750123    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e56b3a3-6b32-4c68-996c-f5a523d07957-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9e56b3a3-6b32-4c68-996c-f5a523d07957" (UID: "9e56b3a3-6b32-4c68-996c-f5a523d07957"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:51:42.851341 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:42.851256    2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e56b3a3-6b32-4c68-996c-f5a523d07957-maas-api-service-ca\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:43.506499 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:43.506467    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch"
Sep 11 21:51:43.506681 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:43.506497    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch" event={"ID":"9e56b3a3-6b32-4c68-996c-f5a523d07957","Type":"ContainerDied","Data":"dfedaf49e3e91b7a987dc030d5b537a1d9287061be56038d72c8ad2f9760d7cc"}
Sep 11 21:51:43.506681 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:43.506554    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfedaf49e3e91b7a987dc030d5b537a1d9287061be56038d72c8ad2f9760d7cc"
Sep 11 21:51:44.419238 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.419201    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:51:44.465849 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.465773    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794b359e-9757-4166-aa5d-f45daae91981-istio-token\") pod \"794b359e-9757-4166-aa5d-f45daae91981\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") "
Sep 11 21:51:44.465849 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.465834    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhqbn\" (UniqueName: \"kubernetes.io/projected/794b359e-9757-4166-aa5d-f45daae91981-kube-api-access-dhqbn\") pod \"794b359e-9757-4166-aa5d-f45daae91981\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") "
Sep 11 21:51:44.466057 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.465883    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-workload-socket\") pod \"794b359e-9757-4166-aa5d-f45daae91981\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") "
Sep 11 21:51:44.466057 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.465917    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794b359e-9757-4166-aa5d-f45daae91981-istio-podinfo\") pod \"794b359e-9757-4166-aa5d-f45daae91981\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") "
Sep 11 21:51:44.466057 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.465934    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-workload-certs\") pod \"794b359e-9757-4166-aa5d-f45daae91981\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") "
Sep 11 21:51:44.466057 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.465983    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-istio-envoy\") pod \"794b359e-9757-4166-aa5d-f45daae91981\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") "
Sep 11 21:51:44.466057 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.466013    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-istio-data\") pod \"794b359e-9757-4166-aa5d-f45daae91981\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") "
Sep 11 21:51:44.466057 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.466048    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794b359e-9757-4166-aa5d-f45daae91981-istiod-ca-cert\") pod \"794b359e-9757-4166-aa5d-f45daae91981\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") "
Sep 11 21:51:44.466418 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.466085    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-credential-socket\") pod \"794b359e-9757-4166-aa5d-f45daae91981\" (UID: \"794b359e-9757-4166-aa5d-f45daae91981\") "
Sep 11 21:51:44.466418 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.466135    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "794b359e-9757-4166-aa5d-f45daae91981" (UID: "794b359e-9757-4166-aa5d-f45daae91981"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:51:44.466418 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.466351    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-istio-data" (OuterVolumeSpecName: "istio-data") pod "794b359e-9757-4166-aa5d-f45daae91981" (UID: "794b359e-9757-4166-aa5d-f45daae91981"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:51:44.466418 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.466393    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "794b359e-9757-4166-aa5d-f45daae91981" (UID: "794b359e-9757-4166-aa5d-f45daae91981"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:51:44.466654 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.466611    2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-workload-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:44.466654 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.466632    2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-workload-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:44.466654 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.466636    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "794b359e-9757-4166-aa5d-f45daae91981" (UID: "794b359e-9757-4166-aa5d-f45daae91981"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:51:44.466811 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.466639    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/794b359e-9757-4166-aa5d-f45daae91981-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "794b359e-9757-4166-aa5d-f45daae91981" (UID: "794b359e-9757-4166-aa5d-f45daae91981"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:51:44.466811 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.466645    2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-istio-data\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:44.467997 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.467974    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/794b359e-9757-4166-aa5d-f45daae91981-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "794b359e-9757-4166-aa5d-f45daae91981" (UID: "794b359e-9757-4166-aa5d-f45daae91981"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:51:44.468140 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.468123    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "794b359e-9757-4166-aa5d-f45daae91981" (UID: "794b359e-9757-4166-aa5d-f45daae91981"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:51:44.468401 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.468367    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794b359e-9757-4166-aa5d-f45daae91981-istio-token" (OuterVolumeSpecName: "istio-token") pod "794b359e-9757-4166-aa5d-f45daae91981" (UID: "794b359e-9757-4166-aa5d-f45daae91981"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:51:44.468485 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.468460    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794b359e-9757-4166-aa5d-f45daae91981-kube-api-access-dhqbn" (OuterVolumeSpecName: "kube-api-access-dhqbn") pod "794b359e-9757-4166-aa5d-f45daae91981" (UID: "794b359e-9757-4166-aa5d-f45daae91981"). InnerVolumeSpecName "kube-api-access-dhqbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:51:44.513838 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.513814    2544 generic.go:356] "Generic (PLEG): container finished" podID="794b359e-9757-4166-aa5d-f45daae91981" containerID="6fb52e4bee67ccd22dcef7009a6b7f5065f47214c36456912c865563981aa4c7" exitCode=0
Sep 11 21:51:44.513949 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.513857    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"
Sep 11 21:51:44.513949 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.513908    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m" event={"ID":"794b359e-9757-4166-aa5d-f45daae91981","Type":"ContainerDied","Data":"6fb52e4bee67ccd22dcef7009a6b7f5065f47214c36456912c865563981aa4c7"}
Sep 11 21:51:44.514024 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.513994    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m" event={"ID":"794b359e-9757-4166-aa5d-f45daae91981","Type":"ContainerDied","Data":"282ec0853e5687f77acc0a0f00bf074d0a6ac0d30da38373feb614e7729ea697"}
Sep 11 21:51:44.514058 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.514022    2544 scope.go:117] "RemoveContainer" containerID="6fb52e4bee67ccd22dcef7009a6b7f5065f47214c36456912c865563981aa4c7"
Sep 11 21:51:44.524934 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.524869    2544 scope.go:117] "RemoveContainer" containerID="5c1ce71bbad49e45020db838b67cd835935c58e773ed1f715f5ed52753f98578"
Sep 11 21:51:44.534009 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.533988    2544 scope.go:117] "RemoveContainer" containerID="6fb52e4bee67ccd22dcef7009a6b7f5065f47214c36456912c865563981aa4c7"
Sep 11 21:51:44.534390 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:51:44.534361    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb52e4bee67ccd22dcef7009a6b7f5065f47214c36456912c865563981aa4c7\": container with ID starting with 6fb52e4bee67ccd22dcef7009a6b7f5065f47214c36456912c865563981aa4c7 not found: ID does not exist" containerID="6fb52e4bee67ccd22dcef7009a6b7f5065f47214c36456912c865563981aa4c7"
Sep 11 21:51:44.534497 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.534399    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb52e4bee67ccd22dcef7009a6b7f5065f47214c36456912c865563981aa4c7"} err="failed to get container status \"6fb52e4bee67ccd22dcef7009a6b7f5065f47214c36456912c865563981aa4c7\": rpc error: code = NotFound desc = could not find container \"6fb52e4bee67ccd22dcef7009a6b7f5065f47214c36456912c865563981aa4c7\": container with ID starting with 6fb52e4bee67ccd22dcef7009a6b7f5065f47214c36456912c865563981aa4c7 not found: ID does not exist"
Sep 11 21:51:44.534497 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.534416    2544 scope.go:117] "RemoveContainer" containerID="5c1ce71bbad49e45020db838b67cd835935c58e773ed1f715f5ed52753f98578"
Sep 11 21:51:44.534695 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:51:44.534677    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c1ce71bbad49e45020db838b67cd835935c58e773ed1f715f5ed52753f98578\": container with ID starting with 5c1ce71bbad49e45020db838b67cd835935c58e773ed1f715f5ed52753f98578 not found: ID does not exist" containerID="5c1ce71bbad49e45020db838b67cd835935c58e773ed1f715f5ed52753f98578"
Sep 11 21:51:44.534740 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.534701    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c1ce71bbad49e45020db838b67cd835935c58e773ed1f715f5ed52753f98578"} err="failed to get container status \"5c1ce71bbad49e45020db838b67cd835935c58e773ed1f715f5ed52753f98578\": rpc error: code = NotFound desc = could not find container \"5c1ce71bbad49e45020db838b67cd835935c58e773ed1f715f5ed52753f98578\": container with ID starting with 5c1ce71bbad49e45020db838b67cd835935c58e773ed1f715f5ed52753f98578 not found: ID does not exist"
Sep 11 21:51:44.536715 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.536695    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"]
Sep 11 21:51:44.541836 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.541817    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e919c5-openshift-default-6765db6bdb-xbh5m"]
Sep 11 21:51:44.567331 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.567312    2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-istio-envoy\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:44.567331 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.567332    2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794b359e-9757-4166-aa5d-f45daae91981-istiod-ca-cert\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:44.567444 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.567341    2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b359e-9757-4166-aa5d-f45daae91981-credential-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:44.567444 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.567350    2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794b359e-9757-4166-aa5d-f45daae91981-istio-token\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:44.567444 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.567359    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhqbn\" (UniqueName: \"kubernetes.io/projected/794b359e-9757-4166-aa5d-f45daae91981-kube-api-access-dhqbn\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:44.567444 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:44.567367    2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794b359e-9757-4166-aa5d-f45daae91981-istio-podinfo\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:45.147547 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:45.147509    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"]
Sep 11 21:51:45.592667 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:45.592633    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="794b359e-9757-4166-aa5d-f45daae91981" path="/var/lib/kubelet/pods/794b359e-9757-4166-aa5d-f45daae91981/volumes"
Sep 11 21:51:49.937586 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:49.937550    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7e6f5e-29819385-2qlm6"]
Sep 11 21:51:49.939616 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:49.939591    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7e6f5e-29819385-2qlm6"]
Sep 11 21:51:49.964879 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:49.964847    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf"]
Sep 11 21:51:49.965164 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:49.965131    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf" podUID="3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908" containerName="payload-pre-processing" containerID="cri-o://72fcfd91c28b90c5176e25b2eba8822f6f659a0ae73b82335440fcfba62eb8a9" gracePeriod=30
Sep 11 21:51:50.127601 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:50.127562    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch"]
Sep 11 21:51:50.130428 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:50.130336    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e6f5e-4pvch"]
Sep 11 21:51:50.227046 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:50.227014    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf"
Sep 11 21:51:50.326662 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:50.326628    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf7kg\" (UniqueName: \"kubernetes.io/projected/3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908-kube-api-access-xf7kg\") pod \"3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908\" (UID: \"3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908\") "
Sep 11 21:51:50.326833 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:50.326739    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908-plugins-config-volume\") pod \"3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908\" (UID: \"3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908\") "
Sep 11 21:51:50.327113 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:50.327086    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908" (UID: "3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:51:50.328775 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:50.328749    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908-kube-api-access-xf7kg" (OuterVolumeSpecName: "kube-api-access-xf7kg") pod "3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908" (UID: "3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908"). InnerVolumeSpecName "kube-api-access-xf7kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:51:50.427742 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:50.427696    2544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908-plugins-config-volume\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:50.427742 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:50.427738    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf7kg\" (UniqueName: \"kubernetes.io/projected/3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908-kube-api-access-xf7kg\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:50.555929 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:50.555894    2544 generic.go:356] "Generic (PLEG): container finished" podID="3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908" containerID="72fcfd91c28b90c5176e25b2eba8822f6f659a0ae73b82335440fcfba62eb8a9" exitCode=0
Sep 11 21:51:50.556083 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:50.555940    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf"
Sep 11 21:51:50.556083 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:50.555988    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf" event={"ID":"3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908","Type":"ContainerDied","Data":"72fcfd91c28b90c5176e25b2eba8822f6f659a0ae73b82335440fcfba62eb8a9"}
Sep 11 21:51:50.556083 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:50.556039    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf" event={"ID":"3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908","Type":"ContainerDied","Data":"5f609eac6ff9e7e20127c8a6858f0c63e5241fa0cfc82a5642b6a4eae0a4a329"}
Sep 11 21:51:50.556083 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:50.556062    2544 scope.go:117] "RemoveContainer" containerID="72fcfd91c28b90c5176e25b2eba8822f6f659a0ae73b82335440fcfba62eb8a9"
Sep 11 21:51:50.565550 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:50.565528    2544 scope.go:117] "RemoveContainer" containerID="72fcfd91c28b90c5176e25b2eba8822f6f659a0ae73b82335440fcfba62eb8a9"
Sep 11 21:51:50.565848 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:51:50.565828    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72fcfd91c28b90c5176e25b2eba8822f6f659a0ae73b82335440fcfba62eb8a9\": container with ID starting with 72fcfd91c28b90c5176e25b2eba8822f6f659a0ae73b82335440fcfba62eb8a9 not found: ID does not exist" containerID="72fcfd91c28b90c5176e25b2eba8822f6f659a0ae73b82335440fcfba62eb8a9"
Sep 11 21:51:50.565919 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:50.565854    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72fcfd91c28b90c5176e25b2eba8822f6f659a0ae73b82335440fcfba62eb8a9"} err="failed to get container status \"72fcfd91c28b90c5176e25b2eba8822f6f659a0ae73b82335440fcfba62eb8a9\": rpc error: code = NotFound desc = could not find container \"72fcfd91c28b90c5176e25b2eba8822f6f659a0ae73b82335440fcfba62eb8a9\": container with ID starting with 72fcfd91c28b90c5176e25b2eba8822f6f659a0ae73b82335440fcfba62eb8a9 not found: ID does not exist"
Sep 11 21:51:50.578656 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:50.578631    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf"]
Sep 11 21:51:50.583198 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:50.583174    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7e6f5e-6cc6467d6b-2zwjf"]
Sep 11 21:51:51.187390 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.187350    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj" podUID="f53b2b57-89dd-4687-ad47-4cddbcbea855" containerName="main" containerID="cri-o://bda028e993fca5b4dcf69f45c906078b05bbc0f521b898c81d4bb9f8a0743112" gracePeriod=45
Sep 11 21:51:51.438704 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.438650    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:51:51.539111 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.539076    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmx9v\" (UniqueName: \"kubernetes.io/projected/f53b2b57-89dd-4687-ad47-4cddbcbea855-kube-api-access-nmx9v\") pod \"f53b2b57-89dd-4687-ad47-4cddbcbea855\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") "
Sep 11 21:51:51.539111 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.539119    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-tmp-dir\") pod \"f53b2b57-89dd-4687-ad47-4cddbcbea855\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") "
Sep 11 21:51:51.539403 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.539156    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-dshm\") pod \"f53b2b57-89dd-4687-ad47-4cddbcbea855\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") "
Sep 11 21:51:51.539403 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.539195    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53b2b57-89dd-4687-ad47-4cddbcbea855-tls-certs\") pod \"f53b2b57-89dd-4687-ad47-4cddbcbea855\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") "
Sep 11 21:51:51.539403 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.539271    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-home\") pod \"f53b2b57-89dd-4687-ad47-4cddbcbea855\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") "
Sep 11 21:51:51.539403 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.539326    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-model-cache\") pod \"f53b2b57-89dd-4687-ad47-4cddbcbea855\" (UID: \"f53b2b57-89dd-4687-ad47-4cddbcbea855\") "
Sep 11 21:51:51.539614 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.539458    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f53b2b57-89dd-4687-ad47-4cddbcbea855" (UID: "f53b2b57-89dd-4687-ad47-4cddbcbea855"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:51:51.539614 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.539555    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-home" (OuterVolumeSpecName: "home") pod "f53b2b57-89dd-4687-ad47-4cddbcbea855" (UID: "f53b2b57-89dd-4687-ad47-4cddbcbea855"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:51:51.539726 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.539603    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-model-cache" (OuterVolumeSpecName: "model-cache") pod "f53b2b57-89dd-4687-ad47-4cddbcbea855" (UID: "f53b2b57-89dd-4687-ad47-4cddbcbea855"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:51:51.539781 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.539729    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:51.539781 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.539744    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:51.539781 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.539752    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:51.541450 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.541429    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f53b2b57-89dd-4687-ad47-4cddbcbea855-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f53b2b57-89dd-4687-ad47-4cddbcbea855" (UID: "f53b2b57-89dd-4687-ad47-4cddbcbea855"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:51:51.541450 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.541439    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53b2b57-89dd-4687-ad47-4cddbcbea855-kube-api-access-nmx9v" (OuterVolumeSpecName: "kube-api-access-nmx9v") pod "f53b2b57-89dd-4687-ad47-4cddbcbea855" (UID: "f53b2b57-89dd-4687-ad47-4cddbcbea855"). InnerVolumeSpecName "kube-api-access-nmx9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:51:51.541588 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.541508    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-dshm" (OuterVolumeSpecName: "dshm") pod "f53b2b57-89dd-4687-ad47-4cddbcbea855" (UID: "f53b2b57-89dd-4687-ad47-4cddbcbea855"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:51:51.563089 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.563058    2544 generic.go:356] "Generic (PLEG): container finished" podID="f53b2b57-89dd-4687-ad47-4cddbcbea855" containerID="bda028e993fca5b4dcf69f45c906078b05bbc0f521b898c81d4bb9f8a0743112" exitCode=0
Sep 11 21:51:51.563240 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.563107    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"
Sep 11 21:51:51.563240 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.563134    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj" event={"ID":"f53b2b57-89dd-4687-ad47-4cddbcbea855","Type":"ContainerDied","Data":"bda028e993fca5b4dcf69f45c906078b05bbc0f521b898c81d4bb9f8a0743112"}
Sep 11 21:51:51.563240 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.563182    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj" event={"ID":"f53b2b57-89dd-4687-ad47-4cddbcbea855","Type":"ContainerDied","Data":"4da90a875034117bc68c6adde6526e310441a3052eed6a2d55b2a7d5805fcf3c"}
Sep 11 21:51:51.563240 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.563200    2544 scope.go:117] "RemoveContainer" containerID="bda028e993fca5b4dcf69f45c906078b05bbc0f521b898c81d4bb9f8a0743112"
Sep 11 21:51:51.573420 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.573396    2544 scope.go:117] "RemoveContainer" containerID="bda028e993fca5b4dcf69f45c906078b05bbc0f521b898c81d4bb9f8a0743112"
Sep 11 21:51:51.573678 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:51:51.573661    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bda028e993fca5b4dcf69f45c906078b05bbc0f521b898c81d4bb9f8a0743112\": container with ID starting with bda028e993fca5b4dcf69f45c906078b05bbc0f521b898c81d4bb9f8a0743112 not found: ID does not exist" containerID="bda028e993fca5b4dcf69f45c906078b05bbc0f521b898c81d4bb9f8a0743112"
Sep 11 21:51:51.573723 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.573685    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bda028e993fca5b4dcf69f45c906078b05bbc0f521b898c81d4bb9f8a0743112"} err="failed to get container status \"bda028e993fca5b4dcf69f45c906078b05bbc0f521b898c81d4bb9f8a0743112\": rpc error: code = NotFound desc = could not find container \"bda028e993fca5b4dcf69f45c906078b05bbc0f521b898c81d4bb9f8a0743112\": container with ID starting with bda028e993fca5b4dcf69f45c906078b05bbc0f521b898c81d4bb9f8a0743112 not found: ID does not exist"
Sep 11 21:51:51.585039 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.585014    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"]
Sep 11 21:51:51.592937 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.592913    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908" path="/var/lib/kubelet/pods/3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908/volumes"
Sep 11 21:51:51.593225 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.593205    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98852096-4bda-4efc-9b23-ebea5edfcbc3" path="/var/lib/kubelet/pods/98852096-4bda-4efc-9b23-ebea5edfcbc3/volumes"
Sep 11 21:51:51.593514 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.593501    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e56b3a3-6b32-4c68-996c-f5a523d07957" path="/var/lib/kubelet/pods/9e56b3a3-6b32-4c68-996c-f5a523d07957/volumes"
Sep 11 21:51:51.593749 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.593737    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7e6f5e/test-model-7e6f5e-kserve-67754f8fc-vwckj"]
Sep 11 21:51:51.641356 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.641309    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmx9v\" (UniqueName: \"kubernetes.io/projected/f53b2b57-89dd-4687-ad47-4cddbcbea855-kube-api-access-nmx9v\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:51.641356 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.641348    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53b2b57-89dd-4687-ad47-4cddbcbea855-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:51.641356 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:51.641363    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53b2b57-89dd-4687-ad47-4cddbcbea855-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:51:53.593343 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:53.593302    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f53b2b57-89dd-4687-ad47-4cddbcbea855" path="/var/lib/kubelet/pods/f53b2b57-89dd-4687-ad47-4cddbcbea855/volumes"
Sep 11 21:51:56.875201 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:56.875163    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"]
Sep 11 21:51:56.875686 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:51:56.875435    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9" podUID="f597fc9d-2f8e-4047-970b-2d85ca4a928d" containerName="istio-proxy" containerID="cri-o://efb2003c4258fd32043c6f0e12417f21f64d8eba06badd307b79e168075c3c14" gracePeriod=30
Sep 11 21:52:00.166485 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.166443    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p" podUID="98a7e7f6-501c-40e0-bf93-27ed97c2ff23" containerName="main" containerID="cri-o://30f037ff19347e808a3d77b04e36c12de7d014b7fa03963eb73755c14d281a92" gracePeriod=45
Sep 11 21:52:00.422353 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.422285    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:52:00.531442 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.531401    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpgwl\" (UniqueName: \"kubernetes.io/projected/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-kube-api-access-hpgwl\") pod \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") "
Sep 11 21:52:00.531442 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.531453    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-model-cache\") pod \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") "
Sep 11 21:52:00.531695 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.531480    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-tls-certs\") pod \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") "
Sep 11 21:52:00.531695 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.531508    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-home\") pod \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") "
Sep 11 21:52:00.531695 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.531530    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-tmp-dir\") pod \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") "
Sep 11 21:52:00.531695 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.531544    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-dshm\") pod \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\" (UID: \"98a7e7f6-501c-40e0-bf93-27ed97c2ff23\") "
Sep 11 21:52:00.531905 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.531837    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-home" (OuterVolumeSpecName: "home") pod "98a7e7f6-501c-40e0-bf93-27ed97c2ff23" (UID: "98a7e7f6-501c-40e0-bf93-27ed97c2ff23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:52:00.531905 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.531849    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-model-cache" (OuterVolumeSpecName: "model-cache") pod "98a7e7f6-501c-40e0-bf93-27ed97c2ff23" (UID: "98a7e7f6-501c-40e0-bf93-27ed97c2ff23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:52:00.531905 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.531863    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98a7e7f6-501c-40e0-bf93-27ed97c2ff23" (UID: "98a7e7f6-501c-40e0-bf93-27ed97c2ff23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:52:00.532046 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.531989    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:00.532046 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.532007    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:00.532046 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.532021    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:00.533753 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.533729    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-dshm" (OuterVolumeSpecName: "dshm") pod "98a7e7f6-501c-40e0-bf93-27ed97c2ff23" (UID: "98a7e7f6-501c-40e0-bf93-27ed97c2ff23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:52:00.533873 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.533854    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98a7e7f6-501c-40e0-bf93-27ed97c2ff23" (UID: "98a7e7f6-501c-40e0-bf93-27ed97c2ff23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:52:00.533922 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.533874    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-kube-api-access-hpgwl" (OuterVolumeSpecName: "kube-api-access-hpgwl") pod "98a7e7f6-501c-40e0-bf93-27ed97c2ff23" (UID: "98a7e7f6-501c-40e0-bf93-27ed97c2ff23"). InnerVolumeSpecName "kube-api-access-hpgwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:52:00.612577 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.612540    2544 generic.go:356] "Generic (PLEG): container finished" podID="98a7e7f6-501c-40e0-bf93-27ed97c2ff23" containerID="30f037ff19347e808a3d77b04e36c12de7d014b7fa03963eb73755c14d281a92" exitCode=0
Sep 11 21:52:00.612577 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.612584    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"
Sep 11 21:52:00.612790 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.612624    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p" event={"ID":"98a7e7f6-501c-40e0-bf93-27ed97c2ff23","Type":"ContainerDied","Data":"30f037ff19347e808a3d77b04e36c12de7d014b7fa03963eb73755c14d281a92"}
Sep 11 21:52:00.612790 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.612666    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p" event={"ID":"98a7e7f6-501c-40e0-bf93-27ed97c2ff23","Type":"ContainerDied","Data":"44339ea3c6e20519cfa1438e53404240b3c3fd48f393343a63d9c7f3a286a827"}
Sep 11 21:52:00.612790 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.612689    2544 scope.go:117] "RemoveContainer" containerID="30f037ff19347e808a3d77b04e36c12de7d014b7fa03963eb73755c14d281a92"
Sep 11 21:52:00.622164 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.622138    2544 scope.go:117] "RemoveContainer" containerID="30f037ff19347e808a3d77b04e36c12de7d014b7fa03963eb73755c14d281a92"
Sep 11 21:52:00.622511 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:52:00.622494    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f037ff19347e808a3d77b04e36c12de7d014b7fa03963eb73755c14d281a92\": container with ID starting with 30f037ff19347e808a3d77b04e36c12de7d014b7fa03963eb73755c14d281a92 not found: ID does not exist" containerID="30f037ff19347e808a3d77b04e36c12de7d014b7fa03963eb73755c14d281a92"
Sep 11 21:52:00.622584 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.622517    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f037ff19347e808a3d77b04e36c12de7d014b7fa03963eb73755c14d281a92"} err="failed to get container status \"30f037ff19347e808a3d77b04e36c12de7d014b7fa03963eb73755c14d281a92\": rpc error: code = NotFound desc = could not find container \"30f037ff19347e808a3d77b04e36c12de7d014b7fa03963eb73755c14d281a92\": container with ID starting with 30f037ff19347e808a3d77b04e36c12de7d014b7fa03963eb73755c14d281a92 not found: ID does not exist"
Sep 11 21:52:00.633162 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.633129    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpgwl\" (UniqueName: \"kubernetes.io/projected/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-kube-api-access-hpgwl\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:00.633364 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.633196    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:00.633364 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.633208    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98a7e7f6-501c-40e0-bf93-27ed97c2ff23-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:00.635958 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.635930    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"]
Sep 11 21:52:00.637917 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:00.637890    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/e2e-conflict-b-09c5f82a-kserve-75f6d7c887-mph6p"]
Sep 11 21:52:01.592050 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:01.592015    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98a7e7f6-501c-40e0-bf93-27ed97c2ff23" path="/var/lib/kubelet/pods/98a7e7f6-501c-40e0-bf93-27ed97c2ff23/volumes"
Sep 11 21:52:02.136998 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.136923    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:52:02.248813 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.248773    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-credential-socket\") pod \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") "
Sep 11 21:52:02.248813 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.248815    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88m5d\" (UniqueName: \"kubernetes.io/projected/f597fc9d-2f8e-4047-970b-2d85ca4a928d-kube-api-access-88m5d\") pod \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") "
Sep 11 21:52:02.249085 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.248853    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-podinfo\") pod \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") "
Sep 11 21:52:02.249085 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.248894    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istiod-ca-cert\") pod \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") "
Sep 11 21:52:02.249085 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.248918    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-workload-certs\") pod \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") "
Sep 11 21:52:02.249270 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.249087    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-workload-socket\") pod \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") "
Sep 11 21:52:02.249270 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.249206    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-data\") pod \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") "
Sep 11 21:52:02.249270 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.249238    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f597fc9d-2f8e-4047-970b-2d85ca4a928d" (UID: "f597fc9d-2f8e-4047-970b-2d85ca4a928d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:52:02.249270 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.249207    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f597fc9d-2f8e-4047-970b-2d85ca4a928d" (UID: "f597fc9d-2f8e-4047-970b-2d85ca4a928d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:52:02.249489 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.249282    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-envoy\") pod \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") "
Sep 11 21:52:02.249489 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.249339    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f597fc9d-2f8e-4047-970b-2d85ca4a928d" (UID: "f597fc9d-2f8e-4047-970b-2d85ca4a928d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:52:02.249489 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.249350    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-token\") pod \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\" (UID: \"f597fc9d-2f8e-4047-970b-2d85ca4a928d\") "
Sep 11 21:52:02.249489 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.249437    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f597fc9d-2f8e-4047-970b-2d85ca4a928d" (UID: "f597fc9d-2f8e-4047-970b-2d85ca4a928d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:52:02.249696 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.249554    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-data" (OuterVolumeSpecName: "istio-data") pod "f597fc9d-2f8e-4047-970b-2d85ca4a928d" (UID: "f597fc9d-2f8e-4047-970b-2d85ca4a928d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:52:02.249833 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.249812    2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-data\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:02.249890 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.249841    2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-credential-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:02.249890 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.249858    2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istiod-ca-cert\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:02.249890 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.249872    2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-workload-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:02.249890 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.249885    2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-workload-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:02.251530 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.251463    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f597fc9d-2f8e-4047-970b-2d85ca4a928d" (UID: "f597fc9d-2f8e-4047-970b-2d85ca4a928d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:52:02.251661 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.251611    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-token" (OuterVolumeSpecName: "istio-token") pod "f597fc9d-2f8e-4047-970b-2d85ca4a928d" (UID: "f597fc9d-2f8e-4047-970b-2d85ca4a928d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:52:02.251773 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.251752    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f597fc9d-2f8e-4047-970b-2d85ca4a928d" (UID: "f597fc9d-2f8e-4047-970b-2d85ca4a928d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:52:02.251887 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.251866    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f597fc9d-2f8e-4047-970b-2d85ca4a928d-kube-api-access-88m5d" (OuterVolumeSpecName: "kube-api-access-88m5d") pod "f597fc9d-2f8e-4047-970b-2d85ca4a928d" (UID: "f597fc9d-2f8e-4047-970b-2d85ca4a928d"). InnerVolumeSpecName "kube-api-access-88m5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:52:02.350728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.350689    2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-token\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:02.350728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.350720    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88m5d\" (UniqueName: \"kubernetes.io/projected/f597fc9d-2f8e-4047-970b-2d85ca4a928d-kube-api-access-88m5d\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:02.350728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.350730    2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-podinfo\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:02.350728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.350739    2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f597fc9d-2f8e-4047-970b-2d85ca4a928d-istio-envoy\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:02.625094 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.625058    2544 generic.go:356] "Generic (PLEG): container finished" podID="f597fc9d-2f8e-4047-970b-2d85ca4a928d" containerID="efb2003c4258fd32043c6f0e12417f21f64d8eba06badd307b79e168075c3c14" exitCode=0
Sep 11 21:52:02.625602 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.625125    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"
Sep 11 21:52:02.625602 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.625141    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9" event={"ID":"f597fc9d-2f8e-4047-970b-2d85ca4a928d","Type":"ContainerDied","Data":"efb2003c4258fd32043c6f0e12417f21f64d8eba06badd307b79e168075c3c14"}
Sep 11 21:52:02.625602 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.625195    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9" event={"ID":"f597fc9d-2f8e-4047-970b-2d85ca4a928d","Type":"ContainerDied","Data":"2c990eabeb84fb41c9cdfc2ffb6f140a40142605566c993cd08170c8357f838f"}
Sep 11 21:52:02.625602 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.625231    2544 scope.go:117] "RemoveContainer" containerID="efb2003c4258fd32043c6f0e12417f21f64d8eba06badd307b79e168075c3c14"
Sep 11 21:52:02.634553 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.634533    2544 scope.go:117] "RemoveContainer" containerID="8622ed6d9b23ae897278a035a57464c8b84a341294ac4b19007db1af147a0607"
Sep 11 21:52:02.643431 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.643409    2544 scope.go:117] "RemoveContainer" containerID="efb2003c4258fd32043c6f0e12417f21f64d8eba06badd307b79e168075c3c14"
Sep 11 21:52:02.643754 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:52:02.643732    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb2003c4258fd32043c6f0e12417f21f64d8eba06badd307b79e168075c3c14\": container with ID starting with efb2003c4258fd32043c6f0e12417f21f64d8eba06badd307b79e168075c3c14 not found: ID does not exist" containerID="efb2003c4258fd32043c6f0e12417f21f64d8eba06badd307b79e168075c3c14"
Sep 11 21:52:02.643825 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.643769    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb2003c4258fd32043c6f0e12417f21f64d8eba06badd307b79e168075c3c14"} err="failed to get container status \"efb2003c4258fd32043c6f0e12417f21f64d8eba06badd307b79e168075c3c14\": rpc error: code = NotFound desc = could not find container \"efb2003c4258fd32043c6f0e12417f21f64d8eba06badd307b79e168075c3c14\": container with ID starting with efb2003c4258fd32043c6f0e12417f21f64d8eba06badd307b79e168075c3c14 not found: ID does not exist"
Sep 11 21:52:02.643825 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.643793    2544 scope.go:117] "RemoveContainer" containerID="8622ed6d9b23ae897278a035a57464c8b84a341294ac4b19007db1af147a0607"
Sep 11 21:52:02.644064 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:52:02.644045    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8622ed6d9b23ae897278a035a57464c8b84a341294ac4b19007db1af147a0607\": container with ID starting with 8622ed6d9b23ae897278a035a57464c8b84a341294ac4b19007db1af147a0607 not found: ID does not exist" containerID="8622ed6d9b23ae897278a035a57464c8b84a341294ac4b19007db1af147a0607"
Sep 11 21:52:02.644108 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.644070    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8622ed6d9b23ae897278a035a57464c8b84a341294ac4b19007db1af147a0607"} err="failed to get container status \"8622ed6d9b23ae897278a035a57464c8b84a341294ac4b19007db1af147a0607\": rpc error: code = NotFound desc = could not find container \"8622ed6d9b23ae897278a035a57464c8b84a341294ac4b19007db1af147a0607\": container with ID starting with 8622ed6d9b23ae897278a035a57464c8b84a341294ac4b19007db1af147a0607 not found: ID does not exist"
Sep 11 21:52:02.647039 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.647017    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"]
Sep 11 21:52:02.651514 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:02.651491    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7e6f5e-openshift-default-cf76475f6-vl7f9"]
Sep 11 21:52:03.593496 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:03.593448    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f597fc9d-2f8e-4047-970b-2d85ca4a928d" path="/var/lib/kubelet/pods/f597fc9d-2f8e-4047-970b-2d85ca4a928d/volumes"
Sep 11 21:52:05.978397 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.978356    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl"]
Sep 11 21:52:05.978868 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.978847    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="794b359e-9757-4166-aa5d-f45daae91981" containerName="istio-proxy"
Sep 11 21:52:05.978956 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.978871    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="794b359e-9757-4166-aa5d-f45daae91981" containerName="istio-proxy"
Sep 11 21:52:05.978956 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.978883    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908" containerName="payload-pre-processing"
Sep 11 21:52:05.978956 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.978892    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908" containerName="payload-pre-processing"
Sep 11 21:52:05.978956 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.978911    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e56b3a3-6b32-4c68-996c-f5a523d07957" containerName="revoke-keys"
Sep 11 21:52:05.978956 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.978920    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e56b3a3-6b32-4c68-996c-f5a523d07957" containerName="revoke-keys"
Sep 11 21:52:05.978956 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.978934    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f53b2b57-89dd-4687-ad47-4cddbcbea855" containerName="main"
Sep 11 21:52:05.978956 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.978943    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="f53b2b57-89dd-4687-ad47-4cddbcbea855" containerName="main"
Sep 11 21:52:05.979323 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.978962    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f597fc9d-2f8e-4047-970b-2d85ca4a928d" containerName="istio-proxy"
Sep 11 21:52:05.979323 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.978970    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="f597fc9d-2f8e-4047-970b-2d85ca4a928d" containerName="istio-proxy"
Sep 11 21:52:05.979323 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.979001    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a7e7f6-501c-40e0-bf93-27ed97c2ff23" containerName="main"
Sep 11 21:52:05.979323 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.979010    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a7e7f6-501c-40e0-bf93-27ed97c2ff23" containerName="main"
Sep 11 21:52:05.979323 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.979100    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="f597fc9d-2f8e-4047-970b-2d85ca4a928d" containerName="istio-proxy"
Sep 11 21:52:05.979323 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.979111    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="794b359e-9757-4166-aa5d-f45daae91981" containerName="istio-proxy"
Sep 11 21:52:05.979323 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.979119    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="98a7e7f6-501c-40e0-bf93-27ed97c2ff23" containerName="main"
Sep 11 21:52:05.979323 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.979130    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d8a68f3-2dbe-4923-a3f8-7fe4e08b6908" containerName="payload-pre-processing"
Sep 11 21:52:05.979323 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.979139    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="f53b2b57-89dd-4687-ad47-4cddbcbea855" containerName="main"
Sep 11 21:52:05.979323 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.979147    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e56b3a3-6b32-4c68-996c-f5a523d07957" containerName="revoke-keys"
Sep 11 21:52:05.979323 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.979155    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="f597fc9d-2f8e-4047-970b-2d85ca4a928d" containerName="istio-proxy"
Sep 11 21:52:05.983810 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.983786    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl"
Sep 11 21:52:05.986787 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.986764    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-7d7821\""
Sep 11 21:52:05.988080 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.988055    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wvqb\" (UniqueName: \"kubernetes.io/projected/9b52ce0f-8e14-4114-9f65-60eb51ec7020-kube-api-access-9wvqb\") pod \"payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl\" (UID: \"9b52ce0f-8e14-4114-9f65-60eb51ec7020\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl"
Sep 11 21:52:05.988197 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.988174    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b52ce0f-8e14-4114-9f65-60eb51ec7020-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl\" (UID: \"9b52ce0f-8e14-4114-9f65-60eb51ec7020\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl"
Sep 11 21:52:05.995964 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:05.995934    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl"]
Sep 11 21:52:06.089001 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:06.088961    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b52ce0f-8e14-4114-9f65-60eb51ec7020-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl\" (UID: \"9b52ce0f-8e14-4114-9f65-60eb51ec7020\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl"
Sep 11 21:52:06.089179 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:06.089081    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wvqb\" (UniqueName: \"kubernetes.io/projected/9b52ce0f-8e14-4114-9f65-60eb51ec7020-kube-api-access-9wvqb\") pod \"payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl\" (UID: \"9b52ce0f-8e14-4114-9f65-60eb51ec7020\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl"
Sep 11 21:52:06.089766 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:06.089736    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b52ce0f-8e14-4114-9f65-60eb51ec7020-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl\" (UID: \"9b52ce0f-8e14-4114-9f65-60eb51ec7020\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl"
Sep 11 21:52:06.097421 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:06.097395    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wvqb\" (UniqueName: \"kubernetes.io/projected/9b52ce0f-8e14-4114-9f65-60eb51ec7020-kube-api-access-9wvqb\") pod \"payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl\" (UID: \"9b52ce0f-8e14-4114-9f65-60eb51ec7020\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl"
Sep 11 21:52:06.293984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:06.293877    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl"
Sep 11 21:52:06.569715 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:06.569684    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl"]
Sep 11 21:52:06.572081 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:52:06.572050    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b52ce0f_8e14_4114_9f65_60eb51ec7020.slice/crio-bc05840e57be4c0151da25f45c7f23bc75044d239b04a7684a442bed426d4370 WatchSource:0}: Error finding container bc05840e57be4c0151da25f45c7f23bc75044d239b04a7684a442bed426d4370: Status 404 returned error can't find the container with id bc05840e57be4c0151da25f45c7f23bc75044d239b04a7684a442bed426d4370
Sep 11 21:52:06.650237 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:06.650189    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl" event={"ID":"9b52ce0f-8e14-4114-9f65-60eb51ec7020","Type":"ContainerStarted","Data":"33aadb6d57a336dbca7fd877f6e34095ffacc68e42782aa9daaffc20a101d4ef"}
Sep 11 21:52:06.650586 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:06.650278    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl" event={"ID":"9b52ce0f-8e14-4114-9f65-60eb51ec7020","Type":"ContainerStarted","Data":"bc05840e57be4c0151da25f45c7f23bc75044d239b04a7684a442bed426d4370"}
Sep 11 21:52:06.650586 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:06.650411    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl"
Sep 11 21:52:06.668198 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:06.668123    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl" podStartSLOduration=1.6681004590000001 podStartE2EDuration="1.668100459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:06.666775469 +0000 UTC m=+1289.636192288" watchObservedRunningTime="2026-09-11 21:52:06.668100459 +0000 UTC m=+1289.637517279"
Sep 11 21:52:13.745096 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.745058    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"]
Sep 11 21:52:13.745612 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.745594    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="794b359e-9757-4166-aa5d-f45daae91981" containerName="istio-proxy"
Sep 11 21:52:13.745657 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.745616    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="794b359e-9757-4166-aa5d-f45daae91981" containerName="istio-proxy"
Sep 11 21:52:13.745657 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.745646    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f597fc9d-2f8e-4047-970b-2d85ca4a928d" containerName="istio-proxy"
Sep 11 21:52:13.745722 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.745655    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="f597fc9d-2f8e-4047-970b-2d85ca4a928d" containerName="istio-proxy"
Sep 11 21:52:13.745769 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.745757    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="794b359e-9757-4166-aa5d-f45daae91981" containerName="istio-proxy"
Sep 11 21:52:13.749268 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.749249    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:13.751924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.751895    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f12741\"/\"distinct-2-w0-f12741-kserve-self-signed-certs\""
Sep 11 21:52:13.756651 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.756611    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"]
Sep 11 21:52:13.854728 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.854685    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-home\") pod \"distinct-2-w0-f12741-kserve-5d7877d57-jv8jg\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") " pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:13.854911 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.854748    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-dshm\") pod \"distinct-2-w0-f12741-kserve-5d7877d57-jv8jg\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") " pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:13.854911 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.854789    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-tls-certs\") pod \"distinct-2-w0-f12741-kserve-5d7877d57-jv8jg\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") " pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:13.854911 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.854807    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf6n9\" (UniqueName: \"kubernetes.io/projected/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-kube-api-access-pf6n9\") pod \"distinct-2-w0-f12741-kserve-5d7877d57-jv8jg\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") " pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:13.854911 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.854831    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-tmp-dir\") pod \"distinct-2-w0-f12741-kserve-5d7877d57-jv8jg\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") " pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:13.855078 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.854936    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-model-cache\") pod \"distinct-2-w0-f12741-kserve-5d7877d57-jv8jg\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") " pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:13.956381 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.956344    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-model-cache\") pod \"distinct-2-w0-f12741-kserve-5d7877d57-jv8jg\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") " pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:13.956597 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.956407    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-home\") pod \"distinct-2-w0-f12741-kserve-5d7877d57-jv8jg\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") " pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:13.956597 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.956440    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-dshm\") pod \"distinct-2-w0-f12741-kserve-5d7877d57-jv8jg\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") " pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:13.956597 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.956462    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-tls-certs\") pod \"distinct-2-w0-f12741-kserve-5d7877d57-jv8jg\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") " pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:13.956597 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.956572    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf6n9\" (UniqueName: \"kubernetes.io/projected/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-kube-api-access-pf6n9\") pod \"distinct-2-w0-f12741-kserve-5d7877d57-jv8jg\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") " pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:13.956837 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.956610    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-tmp-dir\") pod \"distinct-2-w0-f12741-kserve-5d7877d57-jv8jg\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") " pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:13.956897 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.956831    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-model-cache\") pod \"distinct-2-w0-f12741-kserve-5d7877d57-jv8jg\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") " pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:13.956897 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.956861    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-home\") pod \"distinct-2-w0-f12741-kserve-5d7877d57-jv8jg\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") " pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:13.957120 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.957100    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-tmp-dir\") pod \"distinct-2-w0-f12741-kserve-5d7877d57-jv8jg\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") " pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:13.958879 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.958853    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-dshm\") pod \"distinct-2-w0-f12741-kserve-5d7877d57-jv8jg\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") " pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:13.959170 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.959145    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-tls-certs\") pod \"distinct-2-w0-f12741-kserve-5d7877d57-jv8jg\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") " pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:13.967636 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:13.967596    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf6n9\" (UniqueName: \"kubernetes.io/projected/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-kube-api-access-pf6n9\") pod \"distinct-2-w0-f12741-kserve-5d7877d57-jv8jg\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") " pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:14.060467 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:14.060367    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:14.329960 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:14.329933    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"]
Sep 11 21:52:14.332496 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:52:14.332464    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podede62c5f_0b90_49f9_a61d_e5e4209a1a65.slice/crio-011fd26c4760795328fe427216d8c77ae2a9e5f2eaae036d49922ba3cad4a4a0 WatchSource:0}: Error finding container 011fd26c4760795328fe427216d8c77ae2a9e5f2eaae036d49922ba3cad4a4a0: Status 404 returned error can't find the container with id 011fd26c4760795328fe427216d8c77ae2a9e5f2eaae036d49922ba3cad4a4a0
Sep 11 21:52:14.696699 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:14.696605    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg" event={"ID":"ede62c5f-0b90-49f9-a61d-e5e4209a1a65","Type":"ContainerStarted","Data":"f1dca7ba74f817d1fb74ee1e1d5db952b9c9e53ed24cd78614abee42b40e12ba"}
Sep 11 21:52:14.696699 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:14.696649    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg" event={"ID":"ede62c5f-0b90-49f9-a61d-e5e4209a1a65","Type":"ContainerStarted","Data":"011fd26c4760795328fe427216d8c77ae2a9e5f2eaae036d49922ba3cad4a4a0"}
Sep 11 21:52:14.712659 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:14.712600    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg" podStartSLOduration=1.7125779479999999 podStartE2EDuration="1.712577948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:14.711577854 +0000 UTC m=+1297.680994674" watchObservedRunningTime="2026-09-11 21:52:14.712577948 +0000 UTC m=+1297.681994772"
Sep 11 21:52:17.658622 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:17.658590    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl"
Sep 11 21:52:20.400117 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:20.400083    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"]
Sep 11 21:52:24.061133 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:24.061082    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:24.061629 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:24.061153    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:24.078867 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:24.078839    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:24.768208 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:24.768174    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:52:34.745475 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.745427    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"]
Sep 11 21:52:34.749895 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.749865    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:34.752571 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.752539    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-35e777\"/\"distinct-w4-35e777-kserve-self-signed-certs\""
Sep 11 21:52:34.766491 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.760508    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"]
Sep 11 21:52:34.880982 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.880941    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-tls-certs\") pod \"distinct-w4-35e777-kserve-6fd77c5b77-srbn7\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:34.881163 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.881000    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftwp4\" (UniqueName: \"kubernetes.io/projected/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-kube-api-access-ftwp4\") pod \"distinct-w4-35e777-kserve-6fd77c5b77-srbn7\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:34.881163 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.881105    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-tmp-dir\") pod \"distinct-w4-35e777-kserve-6fd77c5b77-srbn7\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:34.881163 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.881155    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-model-cache\") pod \"distinct-w4-35e777-kserve-6fd77c5b77-srbn7\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:34.881306 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.881209    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-home\") pod \"distinct-w4-35e777-kserve-6fd77c5b77-srbn7\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:34.881306 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.881248    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-dshm\") pod \"distinct-w4-35e777-kserve-6fd77c5b77-srbn7\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:34.982099 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.982067    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwp4\" (UniqueName: \"kubernetes.io/projected/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-kube-api-access-ftwp4\") pod \"distinct-w4-35e777-kserve-6fd77c5b77-srbn7\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:34.982323 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.982126    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-tmp-dir\") pod \"distinct-w4-35e777-kserve-6fd77c5b77-srbn7\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:34.982323 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.982172    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-model-cache\") pod \"distinct-w4-35e777-kserve-6fd77c5b77-srbn7\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:34.982323 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.982203    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-home\") pod \"distinct-w4-35e777-kserve-6fd77c5b77-srbn7\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:34.982323 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.982262    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-dshm\") pod \"distinct-w4-35e777-kserve-6fd77c5b77-srbn7\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:34.982323 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.982298    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-tls-certs\") pod \"distinct-w4-35e777-kserve-6fd77c5b77-srbn7\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:34.982612 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.982533    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-tmp-dir\") pod \"distinct-w4-35e777-kserve-6fd77c5b77-srbn7\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:34.982667 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.982613    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-model-cache\") pod \"distinct-w4-35e777-kserve-6fd77c5b77-srbn7\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:34.982667 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.982645    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-home\") pod \"distinct-w4-35e777-kserve-6fd77c5b77-srbn7\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:34.985080 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.985056    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-dshm\") pod \"distinct-w4-35e777-kserve-6fd77c5b77-srbn7\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:34.985243 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.985208    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-tls-certs\") pod \"distinct-w4-35e777-kserve-6fd77c5b77-srbn7\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:34.996054 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:34.995980    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftwp4\" (UniqueName: \"kubernetes.io/projected/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-kube-api-access-ftwp4\") pod \"distinct-w4-35e777-kserve-6fd77c5b77-srbn7\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:35.066817 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.066775    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:35.347147 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.347118    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"]
Sep 11 21:52:35.348704 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:52:35.348675    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbca69aa4_8bb0_43b1_9113_82ad49cd5f6e.slice/crio-04a7b61b796113aaa11a7bd6c75e22de2086ac1ca2a5240055c7895cca2050e6 WatchSource:0}: Error finding container 04a7b61b796113aaa11a7bd6c75e22de2086ac1ca2a5240055c7895cca2050e6: Status 404 returned error can't find the container with id 04a7b61b796113aaa11a7bd6c75e22de2086ac1ca2a5240055c7895cca2050e6
Sep 11 21:52:35.421135 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.421073    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5" podUID="fe047f39-7b76-45c8-8310-7ffbaf6dd38c" containerName="main" containerID="cri-o://055d35fca944f5ef05bb452a6e5d43f8be3dd46991486f248d1bb51c6c963147" gracePeriod=45
Sep 11 21:52:35.658169 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.658143    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:52:35.791903 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.791848    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-tmp-dir\") pod \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") "
Sep 11 21:52:35.791903 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.791906    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-model-cache\") pod \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") "
Sep 11 21:52:35.792445 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.791984    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-home\") pod \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") "
Sep 11 21:52:35.792445 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.792092    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdwld\" (UniqueName: \"kubernetes.io/projected/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-kube-api-access-qdwld\") pod \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") "
Sep 11 21:52:35.792445 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.792116    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe047f39-7b76-45c8-8310-7ffbaf6dd38c" (UID: "fe047f39-7b76-45c8-8310-7ffbaf6dd38c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:52:35.792445 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.792131    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-tls-certs\") pod \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") "
Sep 11 21:52:35.792445 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.792213    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-dshm\") pod \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\" (UID: \"fe047f39-7b76-45c8-8310-7ffbaf6dd38c\") "
Sep 11 21:52:35.792445 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.792261    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe047f39-7b76-45c8-8310-7ffbaf6dd38c" (UID: "fe047f39-7b76-45c8-8310-7ffbaf6dd38c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:52:35.792445 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.792335    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-home" (OuterVolumeSpecName: "home") pod "fe047f39-7b76-45c8-8310-7ffbaf6dd38c" (UID: "fe047f39-7b76-45c8-8310-7ffbaf6dd38c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:52:35.792713 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.792577    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:35.792713 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.792594    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:35.792713 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.792602    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:35.794252 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.794210    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-dshm" (OuterVolumeSpecName: "dshm") pod "fe047f39-7b76-45c8-8310-7ffbaf6dd38c" (UID: "fe047f39-7b76-45c8-8310-7ffbaf6dd38c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:52:35.794373 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.794289    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe047f39-7b76-45c8-8310-7ffbaf6dd38c" (UID: "fe047f39-7b76-45c8-8310-7ffbaf6dd38c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:52:35.794373 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.794345    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-kube-api-access-qdwld" (OuterVolumeSpecName: "kube-api-access-qdwld") pod "fe047f39-7b76-45c8-8310-7ffbaf6dd38c" (UID: "fe047f39-7b76-45c8-8310-7ffbaf6dd38c"). InnerVolumeSpecName "kube-api-access-qdwld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:52:35.820297 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.820255    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7" event={"ID":"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e","Type":"ContainerStarted","Data":"d01f2d4ad6e599b2d69d67b0b7753165f5f1c6263581c1e555bb60376fed5485"}
Sep 11 21:52:35.820483 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.820313    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7" event={"ID":"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e","Type":"ContainerStarted","Data":"04a7b61b796113aaa11a7bd6c75e22de2086ac1ca2a5240055c7895cca2050e6"}
Sep 11 21:52:35.821675 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.821651    2544 generic.go:356] "Generic (PLEG): container finished" podID="fe047f39-7b76-45c8-8310-7ffbaf6dd38c" containerID="055d35fca944f5ef05bb452a6e5d43f8be3dd46991486f248d1bb51c6c963147" exitCode=0
Sep 11 21:52:35.821777 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.821688    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"
Sep 11 21:52:35.821777 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.821744    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5" event={"ID":"fe047f39-7b76-45c8-8310-7ffbaf6dd38c","Type":"ContainerDied","Data":"055d35fca944f5ef05bb452a6e5d43f8be3dd46991486f248d1bb51c6c963147"}
Sep 11 21:52:35.821850 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.821798    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5" event={"ID":"fe047f39-7b76-45c8-8310-7ffbaf6dd38c","Type":"ContainerDied","Data":"d10078f9a2e0383f711fac845713314f462f8cfed71ea06339deff247b3f68f0"}
Sep 11 21:52:35.821850 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.821826    2544 scope.go:117] "RemoveContainer" containerID="055d35fca944f5ef05bb452a6e5d43f8be3dd46991486f248d1bb51c6c963147"
Sep 11 21:52:35.831139 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.831118    2544 scope.go:117] "RemoveContainer" containerID="055d35fca944f5ef05bb452a6e5d43f8be3dd46991486f248d1bb51c6c963147"
Sep 11 21:52:35.831519 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:52:35.831497    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055d35fca944f5ef05bb452a6e5d43f8be3dd46991486f248d1bb51c6c963147\": container with ID starting with 055d35fca944f5ef05bb452a6e5d43f8be3dd46991486f248d1bb51c6c963147 not found: ID does not exist" containerID="055d35fca944f5ef05bb452a6e5d43f8be3dd46991486f248d1bb51c6c963147"
Sep 11 21:52:35.831615 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.831525    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055d35fca944f5ef05bb452a6e5d43f8be3dd46991486f248d1bb51c6c963147"} err="failed to get container status \"055d35fca944f5ef05bb452a6e5d43f8be3dd46991486f248d1bb51c6c963147\": rpc error: code = NotFound desc = could not find container \"055d35fca944f5ef05bb452a6e5d43f8be3dd46991486f248d1bb51c6c963147\": container with ID starting with 055d35fca944f5ef05bb452a6e5d43f8be3dd46991486f248d1bb51c6c963147 not found: ID does not exist"
Sep 11 21:52:35.839988 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.839938    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7" podStartSLOduration=1.839921009 podStartE2EDuration="1.839921009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:35.837969616 +0000 UTC m=+1318.807386439" watchObservedRunningTime="2026-09-11 21:52:35.839921009 +0000 UTC m=+1318.809337829"
Sep 11 21:52:35.852522 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.852488    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"]
Sep 11 21:52:35.856398 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.856369    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/e2e-conflict-a-09c5f82a-kserve-5f49745db9-4rqc5"]
Sep 11 21:52:35.893976 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.893942    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:35.893976 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.893973    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:35.893976 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:35.893983    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdwld\" (UniqueName: \"kubernetes.io/projected/fe047f39-7b76-45c8-8310-7ffbaf6dd38c-kube-api-access-qdwld\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:37.592562 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:37.592532    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe047f39-7b76-45c8-8310-7ffbaf6dd38c" path="/var/lib/kubelet/pods/fe047f39-7b76-45c8-8310-7ffbaf6dd38c/volumes"
Sep 11 21:52:37.686874 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:37.686840    2544 scope.go:117] "RemoveContainer" containerID="fe9d6d54d714c2cbd6c0e5d7fb4ce3ac781f5661ad9a404443bca31b1b50246c"
Sep 11 21:52:37.695020 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:37.695001    2544 scope.go:117] "RemoveContainer" containerID="72d5b3eabb36231e76e75956c5a815ecb457feee93af9e2b4f03e10fabc3eca6"
Sep 11 21:52:37.702496 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:37.702480    2544 scope.go:117] "RemoveContainer" containerID="e0b6b92d57f4772d0aab6b197df8341dac0e710e58c19f79f350af14c91a835b"
Sep 11 21:52:40.431129 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.431092    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf"]
Sep 11 21:52:40.431661 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.431641    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe047f39-7b76-45c8-8310-7ffbaf6dd38c" containerName="main"
Sep 11 21:52:40.431744 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.431665    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe047f39-7b76-45c8-8310-7ffbaf6dd38c" containerName="main"
Sep 11 21:52:40.431801 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.431773    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe047f39-7b76-45c8-8310-7ffbaf6dd38c" containerName="main"
Sep 11 21:52:40.436354 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.436332    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf"
Sep 11 21:52:40.439132 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.439112    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-4e0323-dockercfg-6d24k\""
Sep 11 21:52:40.439259 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.439161    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-4e0323\""
Sep 11 21:52:40.444366 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.444347    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf"]
Sep 11 21:52:40.471122 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.471096    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv"]
Sep 11 21:52:40.474404 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.474389    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv"
Sep 11 21:52:40.486163 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.486144    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv"]
Sep 11 21:52:40.543527 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.543491    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvpzc\" (UniqueName: \"kubernetes.io/projected/01f0581f-2775-45fa-b005-33e7157f81bd-kube-api-access-jvpzc\") pod \"payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf\" (UID: \"01f0581f-2775-45fa-b005-33e7157f81bd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf"
Sep 11 21:52:40.543697 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.543600    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f0581f-2775-45fa-b005-33e7157f81bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf\" (UID: \"01f0581f-2775-45fa-b005-33e7157f81bd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf"
Sep 11 21:52:40.644802 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.644768    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvpzc\" (UniqueName: \"kubernetes.io/projected/01f0581f-2775-45fa-b005-33e7157f81bd-kube-api-access-jvpzc\") pod \"payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf\" (UID: \"01f0581f-2775-45fa-b005-33e7157f81bd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf"
Sep 11 21:52:40.644973 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.644839    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f0581f-2775-45fa-b005-33e7157f81bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf\" (UID: \"01f0581f-2775-45fa-b005-33e7157f81bd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf"
Sep 11 21:52:40.644973 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.644895    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv\" (UID: \"f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv"
Sep 11 21:52:40.644973 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.644923    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbnln\" (UniqueName: \"kubernetes.io/projected/f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6-kube-api-access-pbnln\") pod \"payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv\" (UID: \"f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv"
Sep 11 21:52:40.645633 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.645607    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f0581f-2775-45fa-b005-33e7157f81bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf\" (UID: \"01f0581f-2775-45fa-b005-33e7157f81bd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf"
Sep 11 21:52:40.652730 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.652706    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvpzc\" (UniqueName: \"kubernetes.io/projected/01f0581f-2775-45fa-b005-33e7157f81bd-kube-api-access-jvpzc\") pod \"payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf\" (UID: \"01f0581f-2775-45fa-b005-33e7157f81bd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf"
Sep 11 21:52:40.746418 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.746385    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv\" (UID: \"f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv"
Sep 11 21:52:40.746571 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.746421    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbnln\" (UniqueName: \"kubernetes.io/projected/f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6-kube-api-access-pbnln\") pod \"payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv\" (UID: \"f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv"
Sep 11 21:52:40.746782 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.746758    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf"
Sep 11 21:52:40.747082 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.747061    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv\" (UID: \"f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv"
Sep 11 21:52:40.754058 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.754034    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbnln\" (UniqueName: \"kubernetes.io/projected/f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6-kube-api-access-pbnln\") pod \"payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv\" (UID: \"f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv"
Sep 11 21:52:40.783967 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:40.783937    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv"
Sep 11 21:52:41.019278 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:41.019247    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf"]
Sep 11 21:52:41.020740 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:52:41.020713    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f0581f_2775_45fa_b005_33e7157f81bd.slice/crio-81d16f67d51176f4aa8d4c449faf21e362300b845f763c5ae2e1547d5570a66b WatchSource:0}: Error finding container 81d16f67d51176f4aa8d4c449faf21e362300b845f763c5ae2e1547d5570a66b: Status 404 returned error can't find the container with id 81d16f67d51176f4aa8d4c449faf21e362300b845f763c5ae2e1547d5570a66b
Sep 11 21:52:41.068767 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:41.068740    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv"]
Sep 11 21:52:41.070321 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:52:41.070294    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4bbff87_a75a_4c7f_bb77_f8c612f0b8a6.slice/crio-f3e363b93aba2d4c5d9a976cef26c980c74f29f818a3f3a064c84c70b69bf665 WatchSource:0}: Error finding container f3e363b93aba2d4c5d9a976cef26c980c74f29f818a3f3a064c84c70b69bf665: Status 404 returned error can't find the container with id f3e363b93aba2d4c5d9a976cef26c980c74f29f818a3f3a064c84c70b69bf665
Sep 11 21:52:41.859900 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:41.859866    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv" event={"ID":"f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6","Type":"ContainerStarted","Data":"a3525855837024dd44c4b4871b86a833e5e5d471d62e9ce6626168e78d36d7c1"}
Sep 11 21:52:41.860360 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:41.859923    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv" event={"ID":"f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6","Type":"ContainerStarted","Data":"f3e363b93aba2d4c5d9a976cef26c980c74f29f818a3f3a064c84c70b69bf665"}
Sep 11 21:52:41.860360 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:41.859949    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv"
Sep 11 21:52:41.862092 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:41.862068    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf" event={"ID":"01f0581f-2775-45fa-b005-33e7157f81bd","Type":"ContainerStarted","Data":"c4561ea10adf591dd6b2445123a59a5ebae27e3fd278e3af8f27611cea784708"}
Sep 11 21:52:41.862211 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:41.862115    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf" event={"ID":"01f0581f-2775-45fa-b005-33e7157f81bd","Type":"ContainerStarted","Data":"81d16f67d51176f4aa8d4c449faf21e362300b845f763c5ae2e1547d5570a66b"}
Sep 11 21:52:41.862211 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:41.862141    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf"
Sep 11 21:52:41.876458 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:41.876421    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv" podStartSLOduration=1.8764086500000001 podStartE2EDuration="1.87640865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:41.874944716 +0000 UTC m=+1324.844361537" watchObservedRunningTime="2026-09-11 21:52:41.87640865 +0000 UTC m=+1324.845825469"
Sep 11 21:52:41.889093 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:41.889060    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf" podStartSLOduration=1.889047661 podStartE2EDuration="1.889047661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:41.888765463 +0000 UTC m=+1324.858182283" watchObservedRunningTime="2026-09-11 21:52:41.889047661 +0000 UTC m=+1324.858464477"
Sep 11 21:52:42.935830 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:42.935794    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj"]
Sep 11 21:52:42.939271 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:42.939253    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj"
Sep 11 21:52:42.941577 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:42.941558    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:52:42.941710 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:42.941692    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 21:52:42.948152 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:42.948131    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj"]
Sep 11 21:52:42.966847 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:42.966822    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab289f39-2cea-48e7-b05a-aa145378fe5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj\" (UID: \"ab289f39-2cea-48e7-b05a-aa145378fe5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj"
Sep 11 21:52:43.068163 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:43.068120    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab289f39-2cea-48e7-b05a-aa145378fe5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj\" (UID: \"ab289f39-2cea-48e7-b05a-aa145378fe5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj"
Sep 11 21:52:43.068736 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:43.068715    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab289f39-2cea-48e7-b05a-aa145378fe5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj\" (UID: \"ab289f39-2cea-48e7-b05a-aa145378fe5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj"
Sep 11 21:52:43.253171 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:43.253139    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj"
Sep 11 21:52:43.528035 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:43.528005    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj"]
Sep 11 21:52:43.529240 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:52:43.529198    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab289f39_2cea_48e7_b05a_aa145378fe5a.slice/crio-dfa53bc8d2d9c106f15b0a30b8afd485e42f29c8f214585b5a0beb50193e7ec1 WatchSource:0}: Error finding container dfa53bc8d2d9c106f15b0a30b8afd485e42f29c8f214585b5a0beb50193e7ec1: Status 404 returned error can't find the container with id dfa53bc8d2d9c106f15b0a30b8afd485e42f29c8f214585b5a0beb50193e7ec1
Sep 11 21:52:43.875941 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:43.875849    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj" event={"ID":"ab289f39-2cea-48e7-b05a-aa145378fe5a","Type":"ContainerStarted","Data":"f84622996f3e1081a4d94b2ff219091ddc913446bcd790b0ec92b8539588c38c"}
Sep 11 21:52:43.875941 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:43.875911    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj" event={"ID":"ab289f39-2cea-48e7-b05a-aa145378fe5a","Type":"ContainerStarted","Data":"dfa53bc8d2d9c106f15b0a30b8afd485e42f29c8f214585b5a0beb50193e7ec1"}
Sep 11 21:52:43.891735 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:43.891689    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj" podStartSLOduration=1.891674243 podStartE2EDuration="1.891674243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:43.890098428 +0000 UTC m=+1326.859515243" watchObservedRunningTime="2026-09-11 21:52:43.891674243 +0000 UTC m=+1326.861091063"
Sep 11 21:52:44.882814 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:44.882776    2544 generic.go:356] "Generic (PLEG): container finished" podID="ab289f39-2cea-48e7-b05a-aa145378fe5a" containerID="f84622996f3e1081a4d94b2ff219091ddc913446bcd790b0ec92b8539588c38c" exitCode=0
Sep 11 21:52:44.883300 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:44.882869    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj" event={"ID":"ab289f39-2cea-48e7-b05a-aa145378fe5a","Type":"ContainerDied","Data":"f84622996f3e1081a4d94b2ff219091ddc913446bcd790b0ec92b8539588c38c"}
Sep 11 21:52:45.066887 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:45.066857    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:45.067053 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:45.066906    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:45.079526 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:45.079505    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:45.900361 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:45.900325    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:52:46.031510 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:46.031487    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj"
Sep 11 21:52:46.096881 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:46.096851    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab289f39-2cea-48e7-b05a-aa145378fe5a-maas-api-service-ca\") pod \"ab289f39-2cea-48e7-b05a-aa145378fe5a\" (UID: \"ab289f39-2cea-48e7-b05a-aa145378fe5a\") "
Sep 11 21:52:46.097176 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:46.097156    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab289f39-2cea-48e7-b05a-aa145378fe5a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ab289f39-2cea-48e7-b05a-aa145378fe5a" (UID: "ab289f39-2cea-48e7-b05a-aa145378fe5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:52:46.198369 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:46.198287    2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab289f39-2cea-48e7-b05a-aa145378fe5a-maas-api-service-ca\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:52:46.901661 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:46.901625    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj"
Sep 11 21:52:46.901661 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:46.901648    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj" event={"ID":"ab289f39-2cea-48e7-b05a-aa145378fe5a","Type":"ContainerDied","Data":"dfa53bc8d2d9c106f15b0a30b8afd485e42f29c8f214585b5a0beb50193e7ec1"}
Sep 11 21:52:46.902093 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:46.901715    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfa53bc8d2d9c106f15b0a30b8afd485e42f29c8f214585b5a0beb50193e7ec1"
Sep 11 21:52:52.870738 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:52.870699    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf"
Sep 11 21:52:52.871126 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:52:52.870794    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv"
Sep 11 21:53:00.386638 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.386546    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc"]
Sep 11 21:53:00.387085 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.386803    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc" podUID="03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62" containerName="payload-processing" containerID="cri-o://e5670aed7318c4552e4a95bedf7468c010cd08b72c43d7dcd247cce27781a562" gracePeriod=30
Sep 11 21:53:00.394550 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.394460    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7"]
Sep 11 21:53:00.394733 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.394706    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7" podUID="9ffd593a-a8ee-47c0-b262-7b68a898db62" containerName="payload-pre-processing" containerID="cri-o://2571c772fabaf4b2baeff525a2c5d7ee49d430b0109f5e3d946e75ec26d2c9b9" gracePeriod=30
Sep 11 21:53:00.551270 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.551231    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj"]
Sep 11 21:53:00.555241 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.555200    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f68da5e-zvlzj"]
Sep 11 21:53:00.658888 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.658815    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc"
Sep 11 21:53:00.687833 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.687810    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7"
Sep 11 21:53:00.737908 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.737873    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mll4\" (UniqueName: \"kubernetes.io/projected/9ffd593a-a8ee-47c0-b262-7b68a898db62-kube-api-access-9mll4\") pod \"9ffd593a-a8ee-47c0-b262-7b68a898db62\" (UID: \"9ffd593a-a8ee-47c0-b262-7b68a898db62\") "
Sep 11 21:53:00.738104 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.737942    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffd593a-a8ee-47c0-b262-7b68a898db62-plugins-config-volume\") pod \"9ffd593a-a8ee-47c0-b262-7b68a898db62\" (UID: \"9ffd593a-a8ee-47c0-b262-7b68a898db62\") "
Sep 11 21:53:00.738104 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.737965    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f87q9\" (UniqueName: \"kubernetes.io/projected/03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62-kube-api-access-f87q9\") pod \"03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62\" (UID: \"03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62\") "
Sep 11 21:53:00.738104 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.738018    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62-plugins-config-volume\") pod \"03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62\" (UID: \"03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62\") "
Sep 11 21:53:00.738357 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.738329    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ffd593a-a8ee-47c0-b262-7b68a898db62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ffd593a-a8ee-47c0-b262-7b68a898db62" (UID: "9ffd593a-a8ee-47c0-b262-7b68a898db62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:53:00.738472 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.738448    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62" (UID: "03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:53:00.740075 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.740054    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ffd593a-a8ee-47c0-b262-7b68a898db62-kube-api-access-9mll4" (OuterVolumeSpecName: "kube-api-access-9mll4") pod "9ffd593a-a8ee-47c0-b262-7b68a898db62" (UID: "9ffd593a-a8ee-47c0-b262-7b68a898db62"). InnerVolumeSpecName "kube-api-access-9mll4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:53:00.740147 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.740076    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62-kube-api-access-f87q9" (OuterVolumeSpecName: "kube-api-access-f87q9") pod "03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62" (UID: "03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62"). InnerVolumeSpecName "kube-api-access-f87q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:53:00.839494 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.839438    2544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62-plugins-config-volume\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:53:00.839494 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.839486    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mll4\" (UniqueName: \"kubernetes.io/projected/9ffd593a-a8ee-47c0-b262-7b68a898db62-kube-api-access-9mll4\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:53:00.839494 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.839499    2544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffd593a-a8ee-47c0-b262-7b68a898db62-plugins-config-volume\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:53:00.839738 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.839511    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f87q9\" (UniqueName: \"kubernetes.io/projected/03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62-kube-api-access-f87q9\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:53:00.981675 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.981640    2544 generic.go:356] "Generic (PLEG): container finished" podID="03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62" containerID="e5670aed7318c4552e4a95bedf7468c010cd08b72c43d7dcd247cce27781a562" exitCode=0
Sep 11 21:53:00.981861 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.981690    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc"
Sep 11 21:53:00.981861 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.981729    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc" event={"ID":"03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62","Type":"ContainerDied","Data":"e5670aed7318c4552e4a95bedf7468c010cd08b72c43d7dcd247cce27781a562"}
Sep 11 21:53:00.981861 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.981782    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc" event={"ID":"03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62","Type":"ContainerDied","Data":"8f9c51f578339c1127db84017078a7003137d9200c193d467fa928c4185e7da4"}
Sep 11 21:53:00.981861 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.981806    2544 scope.go:117] "RemoveContainer" containerID="e5670aed7318c4552e4a95bedf7468c010cd08b72c43d7dcd247cce27781a562"
Sep 11 21:53:00.983130 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.983076    2544 generic.go:356] "Generic (PLEG): container finished" podID="9ffd593a-a8ee-47c0-b262-7b68a898db62" containerID="2571c772fabaf4b2baeff525a2c5d7ee49d430b0109f5e3d946e75ec26d2c9b9" exitCode=0
Sep 11 21:53:00.983130 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.983123    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7" event={"ID":"9ffd593a-a8ee-47c0-b262-7b68a898db62","Type":"ContainerDied","Data":"2571c772fabaf4b2baeff525a2c5d7ee49d430b0109f5e3d946e75ec26d2c9b9"}
Sep 11 21:53:00.983336 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.983164    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7" event={"ID":"9ffd593a-a8ee-47c0-b262-7b68a898db62","Type":"ContainerDied","Data":"f2b3f84c23351df3cc98aef370b7eeb9ac0b0e4c2f86d371b533e9939bd82a8c"}
Sep 11 21:53:00.983336 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.983125    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7"
Sep 11 21:53:00.991372 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.991355    2544 scope.go:117] "RemoveContainer" containerID="e5670aed7318c4552e4a95bedf7468c010cd08b72c43d7dcd247cce27781a562"
Sep 11 21:53:00.991696 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:53:00.991636    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5670aed7318c4552e4a95bedf7468c010cd08b72c43d7dcd247cce27781a562\": container with ID starting with e5670aed7318c4552e4a95bedf7468c010cd08b72c43d7dcd247cce27781a562 not found: ID does not exist" containerID="e5670aed7318c4552e4a95bedf7468c010cd08b72c43d7dcd247cce27781a562"
Sep 11 21:53:00.991696 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.991667    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5670aed7318c4552e4a95bedf7468c010cd08b72c43d7dcd247cce27781a562"} err="failed to get container status \"e5670aed7318c4552e4a95bedf7468c010cd08b72c43d7dcd247cce27781a562\": rpc error: code = NotFound desc = could not find container \"e5670aed7318c4552e4a95bedf7468c010cd08b72c43d7dcd247cce27781a562\": container with ID starting with e5670aed7318c4552e4a95bedf7468c010cd08b72c43d7dcd247cce27781a562 not found: ID does not exist"
Sep 11 21:53:00.991696 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:00.991690    2544 scope.go:117] "RemoveContainer" containerID="2571c772fabaf4b2baeff525a2c5d7ee49d430b0109f5e3d946e75ec26d2c9b9"
Sep 11 21:53:01.000282 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:01.000265    2544 scope.go:117] "RemoveContainer" containerID="2571c772fabaf4b2baeff525a2c5d7ee49d430b0109f5e3d946e75ec26d2c9b9"
Sep 11 21:53:01.000559 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:53:01.000540    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2571c772fabaf4b2baeff525a2c5d7ee49d430b0109f5e3d946e75ec26d2c9b9\": container with ID starting with 2571c772fabaf4b2baeff525a2c5d7ee49d430b0109f5e3d946e75ec26d2c9b9 not found: ID does not exist" containerID="2571c772fabaf4b2baeff525a2c5d7ee49d430b0109f5e3d946e75ec26d2c9b9"
Sep 11 21:53:01.000608 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:01.000567    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2571c772fabaf4b2baeff525a2c5d7ee49d430b0109f5e3d946e75ec26d2c9b9"} err="failed to get container status \"2571c772fabaf4b2baeff525a2c5d7ee49d430b0109f5e3d946e75ec26d2c9b9\": rpc error: code = NotFound desc = could not find container \"2571c772fabaf4b2baeff525a2c5d7ee49d430b0109f5e3d946e75ec26d2c9b9\": container with ID starting with 2571c772fabaf4b2baeff525a2c5d7ee49d430b0109f5e3d946e75ec26d2c9b9 not found: ID does not exist"
Sep 11 21:53:01.023829 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:01.023794    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7"]
Sep 11 21:53:01.036070 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:01.036041    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f68da5e-6b96676974-mksf7"]
Sep 11 21:53:01.046488 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:01.046460    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc"]
Sep 11 21:53:01.049008 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:01.048985    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9f68da5e-7d567bf897-n7jbc"]
Sep 11 21:53:01.592212 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:01.592182    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62" path="/var/lib/kubelet/pods/03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62/volumes"
Sep 11 21:53:01.592606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:01.592592    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ffd593a-a8ee-47c0-b262-7b68a898db62" path="/var/lib/kubelet/pods/9ffd593a-a8ee-47c0-b262-7b68a898db62/volumes"
Sep 11 21:53:01.592891 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:01.592881    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab289f39-2cea-48e7-b05a-aa145378fe5a" path="/var/lib/kubelet/pods/ab289f39-2cea-48e7-b05a-aa145378fe5a/volumes"
Sep 11 21:53:26.758647 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.758613    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"]
Sep 11 21:53:26.759192 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.758945    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ffd593a-a8ee-47c0-b262-7b68a898db62" containerName="payload-pre-processing"
Sep 11 21:53:26.759192 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.758955    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ffd593a-a8ee-47c0-b262-7b68a898db62" containerName="payload-pre-processing"
Sep 11 21:53:26.759192 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.758968    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab289f39-2cea-48e7-b05a-aa145378fe5a" containerName="revoke-keys"
Sep 11 21:53:26.759192 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.758973    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab289f39-2cea-48e7-b05a-aa145378fe5a" containerName="revoke-keys"
Sep 11 21:53:26.759192 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.758993    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62" containerName="payload-processing"
Sep 11 21:53:26.759192 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.758998    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62" containerName="payload-processing"
Sep 11 21:53:26.759192 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.759052    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ffd593a-a8ee-47c0-b262-7b68a898db62" containerName="payload-pre-processing"
Sep 11 21:53:26.759192 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.759059    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab289f39-2cea-48e7-b05a-aa145378fe5a" containerName="revoke-keys"
Sep 11 21:53:26.759192 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.759066    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="03df5f5a-9d46-40a1-8eb1-05c6e0ce5f62" containerName="payload-processing"
Sep 11 21:53:26.762531 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.762510    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:26.765810 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.765790    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7d7821\"/\"kube-root-ca.crt\""
Sep 11 21:53:26.765937 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.765871    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7d7821\"/\"auth-test-model-7d7821-kserve-self-signed-certs\""
Sep 11 21:53:26.766011 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.765790    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7d7821\"/\"openshift-service-ca.crt\""
Sep 11 21:53:26.766079 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.765793    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7d7821\"/\"default-dockercfg-bvvhd\""
Sep 11 21:53:26.771806 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.771787    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"]
Sep 11 21:53:26.788359 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.788339    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-tmp-dir\") pod \"auth-test-model-7d7821-kserve-7f459694d7-b845b\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:26.788475 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.788366    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v84d\" (UniqueName: \"kubernetes.io/projected/3882d1cd-226a-4def-a263-888579879ff6-kube-api-access-9v84d\") pod \"auth-test-model-7d7821-kserve-7f459694d7-b845b\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:26.788475 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.788387    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-model-cache\") pod \"auth-test-model-7d7821-kserve-7f459694d7-b845b\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:26.788475 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.788408    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-home\") pod \"auth-test-model-7d7821-kserve-7f459694d7-b845b\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:26.788653 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.788516    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-dshm\") pod \"auth-test-model-7d7821-kserve-7f459694d7-b845b\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:26.788653 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.788556    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3882d1cd-226a-4def-a263-888579879ff6-tls-certs\") pod \"auth-test-model-7d7821-kserve-7f459694d7-b845b\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:26.889456 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.889425    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-model-cache\") pod \"auth-test-model-7d7821-kserve-7f459694d7-b845b\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:26.889582 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.889463    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-home\") pod \"auth-test-model-7d7821-kserve-7f459694d7-b845b\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:26.889582 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.889497    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-dshm\") pod \"auth-test-model-7d7821-kserve-7f459694d7-b845b\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:26.889582 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.889517    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3882d1cd-226a-4def-a263-888579879ff6-tls-certs\") pod \"auth-test-model-7d7821-kserve-7f459694d7-b845b\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:26.889724 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.889595    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-tmp-dir\") pod \"auth-test-model-7d7821-kserve-7f459694d7-b845b\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:26.889724 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.889612    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v84d\" (UniqueName: \"kubernetes.io/projected/3882d1cd-226a-4def-a263-888579879ff6-kube-api-access-9v84d\") pod \"auth-test-model-7d7821-kserve-7f459694d7-b845b\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:26.889861 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.889836    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-model-cache\") pod \"auth-test-model-7d7821-kserve-7f459694d7-b845b\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:26.889861 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.889852    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-home\") pod \"auth-test-model-7d7821-kserve-7f459694d7-b845b\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:26.889980 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.889961    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-tmp-dir\") pod \"auth-test-model-7d7821-kserve-7f459694d7-b845b\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:26.891749 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.891729    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-dshm\") pod \"auth-test-model-7d7821-kserve-7f459694d7-b845b\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:26.891996 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.891980    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3882d1cd-226a-4def-a263-888579879ff6-tls-certs\") pod \"auth-test-model-7d7821-kserve-7f459694d7-b845b\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:26.896723 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:26.896703    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v84d\" (UniqueName: \"kubernetes.io/projected/3882d1cd-226a-4def-a263-888579879ff6-kube-api-access-9v84d\") pod \"auth-test-model-7d7821-kserve-7f459694d7-b845b\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:27.072307 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:27.072184    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:27.343020 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:27.342994    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"]
Sep 11 21:53:27.344607 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:53:27.344583    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3882d1cd_226a_4def_a263_888579879ff6.slice/crio-c2d0dc5e2b8ef162526ceaed95c21d9c45b9604bf0b07a0dbd20f8a177810d7e WatchSource:0}: Error finding container c2d0dc5e2b8ef162526ceaed95c21d9c45b9604bf0b07a0dbd20f8a177810d7e: Status 404 returned error can't find the container with id c2d0dc5e2b8ef162526ceaed95c21d9c45b9604bf0b07a0dbd20f8a177810d7e
Sep 11 21:53:28.124071 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:28.124039    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b" event={"ID":"3882d1cd-226a-4def-a263-888579879ff6","Type":"ContainerStarted","Data":"2c2437a3261adb652eb740abf51680fa2f17c802e7797eb140366d27d84aea18"}
Sep 11 21:53:28.124479 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:28.124081    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b" event={"ID":"3882d1cd-226a-4def-a263-888579879ff6","Type":"ContainerStarted","Data":"c2d0dc5e2b8ef162526ceaed95c21d9c45b9604bf0b07a0dbd20f8a177810d7e"}
Sep 11 21:53:28.140679 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:28.140631    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b" podStartSLOduration=2.140617904 podStartE2EDuration="2.140617904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:28.13933802 +0000 UTC m=+1371.108754862" watchObservedRunningTime="2026-09-11 21:53:28.140617904 +0000 UTC m=+1371.110034761"
Sep 11 21:53:37.072935 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:37.072897    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:37.073425 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:37.072947    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:37.085345 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:37.085318    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:37.180820 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:37.180791    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:53:45.045793 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.045751    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"]
Sep 11 21:53:45.050431 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.050407    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.052700 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.052681    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-35e777\"/\"distinct-2-w4-35e777-kserve-self-signed-certs\""
Sep 11 21:53:45.056898 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.056873    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"]
Sep 11 21:53:45.156366 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.156339    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17a22aae-0368-455b-9d83-308bc9760f26-tls-certs\") pod \"distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.156500 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.156391    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-dshm\") pod \"distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.156500 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.156459    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-tmp-dir\") pod \"distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.156573 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.156512    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68hf2\" (UniqueName: \"kubernetes.io/projected/17a22aae-0368-455b-9d83-308bc9760f26-kube-api-access-68hf2\") pod \"distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.156573 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.156544    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-home\") pod \"distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.156573 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.156564    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-model-cache\") pod \"distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.257425 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.257397    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-dshm\") pod \"distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.257569 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.257434    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-tmp-dir\") pod \"distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.257569 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.257479    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68hf2\" (UniqueName: \"kubernetes.io/projected/17a22aae-0368-455b-9d83-308bc9760f26-kube-api-access-68hf2\") pod \"distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.257569 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.257505    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-home\") pod \"distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.257569 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.257525    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-model-cache\") pod \"distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.257762 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.257576    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17a22aae-0368-455b-9d83-308bc9760f26-tls-certs\") pod \"distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.257890 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.257869    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-tmp-dir\") pod \"distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.257959 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.257885    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-home\") pod \"distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.258016 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.257966    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-model-cache\") pod \"distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.259589 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.259570    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-dshm\") pod \"distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.259920 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.259904    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17a22aae-0368-455b-9d83-308bc9760f26-tls-certs\") pod \"distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.264785 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.264763    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68hf2\" (UniqueName: \"kubernetes.io/projected/17a22aae-0368-455b-9d83-308bc9760f26-kube-api-access-68hf2\") pod \"distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") " pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.360943 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.360855    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:45.638106 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:45.638075    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"]
Sep 11 21:53:45.639317 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:53:45.639285    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17a22aae_0368_455b_9d83_308bc9760f26.slice/crio-5ce0a4f01166cf4ccc145772414640c44d91fb7d3816878939d673df9108292b WatchSource:0}: Error finding container 5ce0a4f01166cf4ccc145772414640c44d91fb7d3816878939d673df9108292b: Status 404 returned error can't find the container with id 5ce0a4f01166cf4ccc145772414640c44d91fb7d3816878939d673df9108292b
Sep 11 21:53:46.215795 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:46.215760    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r" event={"ID":"17a22aae-0368-455b-9d83-308bc9760f26","Type":"ContainerStarted","Data":"da62f719acf07490586bbd29ecb1e1b87190ad21f739445c7e8d709227633647"}
Sep 11 21:53:46.215795 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:46.215802    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r" event={"ID":"17a22aae-0368-455b-9d83-308bc9760f26","Type":"ContainerStarted","Data":"5ce0a4f01166cf4ccc145772414640c44d91fb7d3816878939d673df9108292b"}
Sep 11 21:53:46.237716 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:46.237667    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r" podStartSLOduration=1.237648087 podStartE2EDuration="1.237648087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:46.235904109 +0000 UTC m=+1389.205320930" watchObservedRunningTime="2026-09-11 21:53:46.237648087 +0000 UTC m=+1389.207064909"
Sep 11 21:53:55.361148 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:55.361100    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:55.361148 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:55.361160    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:55.373456 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:55.373427    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:53:56.278831 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:53:56.278805    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:54:37.772907 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:54:37.772833    2544 scope.go:117] "RemoveContainer" containerID="f018e6f3f6b282bc00ce9e50aff16df1b70d04f59145f9d1ea3a2cccfe285e96"
Sep 11 21:54:55.228521 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:54:55.228484    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94"]
Sep 11 21:54:55.233356 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:54:55.233322    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94"
Sep 11 21:54:55.235694 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:54:55.235673    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1132e5\""
Sep 11 21:54:55.235817 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:54:55.235760    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1132e5-dockercfg-8q95d\""
Sep 11 21:54:55.249019 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:54:55.244918    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94"]
Sep 11 21:54:55.351278 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:54:55.351213    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd7j9\" (UniqueName: \"kubernetes.io/projected/d9311bc7-5cb2-4261-9084-43b77f0f8402-kube-api-access-rd7j9\") pod \"payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94\" (UID: \"d9311bc7-5cb2-4261-9084-43b77f0f8402\") " pod="openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94"
Sep 11 21:54:55.351453 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:54:55.351291    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9311bc7-5cb2-4261-9084-43b77f0f8402-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94\" (UID: \"d9311bc7-5cb2-4261-9084-43b77f0f8402\") " pod="openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94"
Sep 11 21:54:55.452419 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:54:55.452385    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd7j9\" (UniqueName: \"kubernetes.io/projected/d9311bc7-5cb2-4261-9084-43b77f0f8402-kube-api-access-rd7j9\") pod \"payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94\" (UID: \"d9311bc7-5cb2-4261-9084-43b77f0f8402\") " pod="openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94"
Sep 11 21:54:55.452607 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:54:55.452430    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9311bc7-5cb2-4261-9084-43b77f0f8402-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94\" (UID: \"d9311bc7-5cb2-4261-9084-43b77f0f8402\") " pod="openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94"
Sep 11 21:54:55.453075 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:54:55.453057    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9311bc7-5cb2-4261-9084-43b77f0f8402-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94\" (UID: \"d9311bc7-5cb2-4261-9084-43b77f0f8402\") " pod="openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94"
Sep 11 21:54:55.459730 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:54:55.459707    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd7j9\" (UniqueName: \"kubernetes.io/projected/d9311bc7-5cb2-4261-9084-43b77f0f8402-kube-api-access-rd7j9\") pod \"payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94\" (UID: \"d9311bc7-5cb2-4261-9084-43b77f0f8402\") " pod="openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94"
Sep 11 21:54:55.545075 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:54:55.544979    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94"
Sep 11 21:54:55.816999 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:54:55.816972    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94"]
Sep 11 21:54:56.574189 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:54:56.574155    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94" event={"ID":"d9311bc7-5cb2-4261-9084-43b77f0f8402","Type":"ContainerStarted","Data":"c964f1ada1236b50a33f57e4bf140f2eb5467e8b69b8a1277d90d636ec9a1310"}
Sep 11 21:54:56.574594 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:54:56.574214    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94" event={"ID":"d9311bc7-5cb2-4261-9084-43b77f0f8402","Type":"ContainerStarted","Data":"78809314bf4b64e1ecc6b8e601f100fe27c904535a44b9c3a77f95c0df8574b5"}
Sep 11 21:54:56.574594 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:54:56.574353    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94"
Sep 11 21:54:56.594561 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:54:56.594516    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94" podStartSLOduration=1.594502201 podStartE2EDuration="1.594502201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:54:56.592695862 +0000 UTC m=+1459.562112682" watchObservedRunningTime="2026-09-11 21:54:56.594502201 +0000 UTC m=+1459.563919019"
Sep 11 21:55:07.581414 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:07.581381    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94"
Sep 11 21:55:20.578402 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.578365    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"]
Sep 11 21:55:20.581914 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.581890    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.584527 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.584487    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-24efea-openshift-default-dockercfg-64f6x\""
Sep 11 21:55:20.593479 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.593457    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"]
Sep 11 21:55:20.699353 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.699235    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e595e7-a303-4af1-b7ba-cd43a4407b99-istiod-ca-cert\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.699353 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.699342    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-data\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.722497 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.699397    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nfd9\" (UniqueName: \"kubernetes.io/projected/40e595e7-a303-4af1-b7ba-cd43a4407b99-kube-api-access-8nfd9\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.722497 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.699428    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-workload-socket\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.722497 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.699445    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-token\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.722497 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.699473    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-workload-certs\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.722497 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.699514    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-podinfo\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.722497 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.699560    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-credential-socket\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.722497 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.699580    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-envoy\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.800934 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.800899    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-data\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.801122 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.800952    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nfd9\" (UniqueName: \"kubernetes.io/projected/40e595e7-a303-4af1-b7ba-cd43a4407b99-kube-api-access-8nfd9\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.801122 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.800975    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-workload-socket\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.801122 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.800992    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-token\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.801122 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.801020    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-workload-certs\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.801122 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.801037    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-podinfo\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.801122 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.801065    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-credential-socket\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.801122 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.801089    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-envoy\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.801533 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.801169    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e595e7-a303-4af1-b7ba-cd43a4407b99-istiod-ca-cert\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.801533 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.801389    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-data\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.801650 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.801567    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-workload-certs\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.801650 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.801592    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-workload-socket\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.801650 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.801616    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-credential-socket\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.801954 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.801935    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e595e7-a303-4af1-b7ba-cd43a4407b99-istiod-ca-cert\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.803324 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.803305    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-envoy\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.803467 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.803450    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-podinfo\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.809068 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.809047    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-token\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.809197 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.809078    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nfd9\" (UniqueName: \"kubernetes.io/projected/40e595e7-a303-4af1-b7ba-cd43a4407b99-kube-api-access-8nfd9\") pod \"e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") " pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:20.892700 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:20.892624    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:21.178267 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:21.178239    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"]
Sep 11 21:55:21.179532 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:55:21.179504    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e595e7_a303_4af1_b7ba_cd43a4407b99.slice/crio-0b07f8c29709a261b4e6a6f80231462596a9bf927efd0422642b2e63d37a7219 WatchSource:0}: Error finding container 0b07f8c29709a261b4e6a6f80231462596a9bf927efd0422642b2e63d37a7219: Status 404 returned error can't find the container with id 0b07f8c29709a261b4e6a6f80231462596a9bf927efd0422642b2e63d37a7219
Sep 11 21:55:21.181707 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:21.181669    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:55:21.181876 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:21.181749    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:55:21.181876 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:21.181792    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:55:21.706358 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:21.706321    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk" event={"ID":"40e595e7-a303-4af1-b7ba-cd43a4407b99","Type":"ContainerStarted","Data":"3623ffa890d60d48603a107cad9f214f0274966c7f3a98dd4fc56bb01ff39a4a"}
Sep 11 21:55:21.706738 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:21.706383    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk" event={"ID":"40e595e7-a303-4af1-b7ba-cd43a4407b99","Type":"ContainerStarted","Data":"0b07f8c29709a261b4e6a6f80231462596a9bf927efd0422642b2e63d37a7219"}
Sep 11 21:55:21.728555 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:21.728509    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk" podStartSLOduration=1.7284957859999999 podStartE2EDuration="1.728495786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:55:21.725783754 +0000 UTC m=+1484.695200590" watchObservedRunningTime="2026-09-11 21:55:21.728495786 +0000 UTC m=+1484.697912605"
Sep 11 21:55:21.893152 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:21.893120    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:22.898285 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:22.898259    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:23.716886 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:23.716857    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:23.717817 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:23.717793    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:55:37.791671 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:37.791636    2544 scope.go:117] "RemoveContainer" containerID="f1b0da04ab3af3ad60eb58f535c6298059eac71bcf2745eab15457d114592158"
Sep 11 21:55:56.616521 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:56.616481    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5"]
Sep 11 21:55:56.627163 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:56.627129    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5"
Sep 11 21:55:56.627763 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:56.627737    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5"]
Sep 11 21:55:56.629589 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:56.629568    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:55:56.629713 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:56.629694    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 21:55:56.753941 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:56.753894    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291346f6-c663-433d-b065-6f3564dfd387-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5\" (UID: \"291346f6-c663-433d-b065-6f3564dfd387\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5"
Sep 11 21:55:56.855174 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:56.855136    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291346f6-c663-433d-b065-6f3564dfd387-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5\" (UID: \"291346f6-c663-433d-b065-6f3564dfd387\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5"
Sep 11 21:55:56.855822 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:56.855795    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291346f6-c663-433d-b065-6f3564dfd387-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5\" (UID: \"291346f6-c663-433d-b065-6f3564dfd387\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5"
Sep 11 21:55:56.940793 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:56.940707    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5"
Sep 11 21:55:57.230507 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:57.230482    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5"]
Sep 11 21:55:57.233152 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:55:57.233119    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291346f6_c663_433d_b065_6f3564dfd387.slice/crio-ca2ccf41f0ac651a9dcbcbc5acebc06a0585c4ae0d8ac7ef9274bffbaa81561e WatchSource:0}: Error finding container ca2ccf41f0ac651a9dcbcbc5acebc06a0585c4ae0d8ac7ef9274bffbaa81561e: Status 404 returned error can't find the container with id ca2ccf41f0ac651a9dcbcbc5acebc06a0585c4ae0d8ac7ef9274bffbaa81561e
Sep 11 21:55:57.895016 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:57.894916    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5" event={"ID":"291346f6-c663-433d-b065-6f3564dfd387","Type":"ContainerStarted","Data":"92c1caf7863486efdbd921eac72e3a454d394584f8def156dab28ceb405c5735"}
Sep 11 21:55:57.895016 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:57.894985    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5" event={"ID":"291346f6-c663-433d-b065-6f3564dfd387","Type":"ContainerStarted","Data":"ca2ccf41f0ac651a9dcbcbc5acebc06a0585c4ae0d8ac7ef9274bffbaa81561e"}
Sep 11 21:55:57.910468 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:57.910416    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5" podStartSLOduration=1.9103999379999999 podStartE2EDuration="1.910399938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:55:57.909484547 +0000 UTC m=+1520.878901366" watchObservedRunningTime="2026-09-11 21:55:57.910399938 +0000 UTC m=+1520.879816751"
Sep 11 21:55:58.901685 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:58.901655    2544 generic.go:356] "Generic (PLEG): container finished" podID="291346f6-c663-433d-b065-6f3564dfd387" containerID="92c1caf7863486efdbd921eac72e3a454d394584f8def156dab28ceb405c5735" exitCode=0
Sep 11 21:55:58.902069 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:55:58.901710    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5" event={"ID":"291346f6-c663-433d-b065-6f3564dfd387","Type":"ContainerDied","Data":"92c1caf7863486efdbd921eac72e3a454d394584f8def156dab28ceb405c5735"}
Sep 11 21:56:00.038247 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:00.038210    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5"
Sep 11 21:56:00.087302 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:00.087267    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291346f6-c663-433d-b065-6f3564dfd387-maas-api-service-ca\") pod \"291346f6-c663-433d-b065-6f3564dfd387\" (UID: \"291346f6-c663-433d-b065-6f3564dfd387\") "
Sep 11 21:56:00.087726 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:00.087697    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291346f6-c663-433d-b065-6f3564dfd387-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "291346f6-c663-433d-b065-6f3564dfd387" (UID: "291346f6-c663-433d-b065-6f3564dfd387"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:56:00.188356 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:00.188265    2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291346f6-c663-433d-b065-6f3564dfd387-maas-api-service-ca\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:00.913087 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:00.913054    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5" event={"ID":"291346f6-c663-433d-b065-6f3564dfd387","Type":"ContainerDied","Data":"ca2ccf41f0ac651a9dcbcbc5acebc06a0585c4ae0d8ac7ef9274bffbaa81561e"}
Sep 11 21:56:00.913285 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:00.913104    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca2ccf41f0ac651a9dcbcbc5acebc06a0585c4ae0d8ac7ef9274bffbaa81561e"
Sep 11 21:56:00.913285 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:00.913104    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5"
Sep 11 21:56:11.983244 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:11.983199    2544 generic.go:356] "Generic (PLEG): container finished" podID="40e595e7-a303-4af1-b7ba-cd43a4407b99" containerID="3623ffa890d60d48603a107cad9f214f0274966c7f3a98dd4fc56bb01ff39a4a" exitCode=0
Sep 11 21:56:11.983634 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:11.983276    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk" event={"ID":"40e595e7-a303-4af1-b7ba-cd43a4407b99","Type":"ContainerDied","Data":"3623ffa890d60d48603a107cad9f214f0274966c7f3a98dd4fc56bb01ff39a4a"}
Sep 11 21:56:11.983726 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:11.983712    2544 scope.go:117] "RemoveContainer" containerID="3623ffa890d60d48603a107cad9f214f0274966c7f3a98dd4fc56bb01ff39a4a"
Sep 11 21:56:11.984899 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:11.984870    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:56:11.984997 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:11.984939    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:56:11.984997 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:11.984969    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 21:56:12.480953 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:12.480919    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94"]
Sep 11 21:56:12.481150 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:12.481124    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94" podUID="d9311bc7-5cb2-4261-9084-43b77f0f8402" containerName="payload-processing" containerID="cri-o://c964f1ada1236b50a33f57e4bf140f2eb5467e8b69b8a1277d90d636ec9a1310" gracePeriod=30
Sep 11 21:56:12.646526 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:12.646489    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5"]
Sep 11 21:56:12.654551 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:12.654516    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1132e5-lhcb5"]
Sep 11 21:56:12.750381 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:12.750273    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94"
Sep 11 21:56:12.918591 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:12.918554    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd7j9\" (UniqueName: \"kubernetes.io/projected/d9311bc7-5cb2-4261-9084-43b77f0f8402-kube-api-access-rd7j9\") pod \"d9311bc7-5cb2-4261-9084-43b77f0f8402\" (UID: \"d9311bc7-5cb2-4261-9084-43b77f0f8402\") "
Sep 11 21:56:12.918778 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:12.918598    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9311bc7-5cb2-4261-9084-43b77f0f8402-plugins-config-volume\") pod \"d9311bc7-5cb2-4261-9084-43b77f0f8402\" (UID: \"d9311bc7-5cb2-4261-9084-43b77f0f8402\") "
Sep 11 21:56:12.918990 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:12.918963    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9311bc7-5cb2-4261-9084-43b77f0f8402-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d9311bc7-5cb2-4261-9084-43b77f0f8402" (UID: "d9311bc7-5cb2-4261-9084-43b77f0f8402"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:56:12.920679 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:12.920657    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9311bc7-5cb2-4261-9084-43b77f0f8402-kube-api-access-rd7j9" (OuterVolumeSpecName: "kube-api-access-rd7j9") pod "d9311bc7-5cb2-4261-9084-43b77f0f8402" (UID: "d9311bc7-5cb2-4261-9084-43b77f0f8402"). InnerVolumeSpecName "kube-api-access-rd7j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:12.990496 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:12.990457    2544 generic.go:356] "Generic (PLEG): container finished" podID="d9311bc7-5cb2-4261-9084-43b77f0f8402" containerID="c964f1ada1236b50a33f57e4bf140f2eb5467e8b69b8a1277d90d636ec9a1310" exitCode=0
Sep 11 21:56:12.990973 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:12.990519    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94"
Sep 11 21:56:12.990973 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:12.990548    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94" event={"ID":"d9311bc7-5cb2-4261-9084-43b77f0f8402","Type":"ContainerDied","Data":"c964f1ada1236b50a33f57e4bf140f2eb5467e8b69b8a1277d90d636ec9a1310"}
Sep 11 21:56:12.990973 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:12.990650    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94" event={"ID":"d9311bc7-5cb2-4261-9084-43b77f0f8402","Type":"ContainerDied","Data":"78809314bf4b64e1ecc6b8e601f100fe27c904535a44b9c3a77f95c0df8574b5"}
Sep 11 21:56:12.990973 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:12.990683    2544 scope.go:117] "RemoveContainer" containerID="c964f1ada1236b50a33f57e4bf140f2eb5467e8b69b8a1277d90d636ec9a1310"
Sep 11 21:56:12.992763 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:12.992735    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk" event={"ID":"40e595e7-a303-4af1-b7ba-cd43a4407b99","Type":"ContainerStarted","Data":"6bab0200d462cc8844c469e373b5e5ba7776c817102d74b65ee7e0f34765d6f4"}
Sep 11 21:56:13.000059 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:13.000041    2544 scope.go:117] "RemoveContainer" containerID="c964f1ada1236b50a33f57e4bf140f2eb5467e8b69b8a1277d90d636ec9a1310"
Sep 11 21:56:13.000345 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:56:13.000327    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c964f1ada1236b50a33f57e4bf140f2eb5467e8b69b8a1277d90d636ec9a1310\": container with ID starting with c964f1ada1236b50a33f57e4bf140f2eb5467e8b69b8a1277d90d636ec9a1310 not found: ID does not exist" containerID="c964f1ada1236b50a33f57e4bf140f2eb5467e8b69b8a1277d90d636ec9a1310"
Sep 11 21:56:13.000420 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:13.000354    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c964f1ada1236b50a33f57e4bf140f2eb5467e8b69b8a1277d90d636ec9a1310"} err="failed to get container status \"c964f1ada1236b50a33f57e4bf140f2eb5467e8b69b8a1277d90d636ec9a1310\": rpc error: code = NotFound desc = could not find container \"c964f1ada1236b50a33f57e4bf140f2eb5467e8b69b8a1277d90d636ec9a1310\": container with ID starting with c964f1ada1236b50a33f57e4bf140f2eb5467e8b69b8a1277d90d636ec9a1310 not found: ID does not exist"
Sep 11 21:56:13.019707 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:13.019684    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd7j9\" (UniqueName: \"kubernetes.io/projected/d9311bc7-5cb2-4261-9084-43b77f0f8402-kube-api-access-rd7j9\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:13.019707 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:13.019704    2544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9311bc7-5cb2-4261-9084-43b77f0f8402-plugins-config-volume\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:13.027741 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:13.027530    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94"]
Sep 11 21:56:13.033315 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:13.033271    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1132e5-7f4bbdf5f7-bdm94"]
Sep 11 21:56:13.593489 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:13.593452    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="291346f6-c663-433d-b065-6f3564dfd387" path="/var/lib/kubelet/pods/291346f6-c663-433d-b065-6f3564dfd387/volumes"
Sep 11 21:56:13.593856 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:13.593843    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9311bc7-5cb2-4261-9084-43b77f0f8402" path="/var/lib/kubelet/pods/d9311bc7-5cb2-4261-9084-43b77f0f8402/volumes"
Sep 11 21:56:13.893688 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:13.893612    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:56:13.893835 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:13.893723    2544 patch_prober.go:28] interesting pod/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 11 21:56:13.893835 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:13.893778    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk" podUID="40e595e7-a303-4af1-b7ba-cd43a4407b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 11 21:56:14.893471 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:14.893435    2544 patch_prober.go:28] interesting pod/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 11 21:56:14.893867 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:14.893516    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk" podUID="40e595e7-a303-4af1-b7ba-cd43a4407b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 11 21:56:15.893508 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:15.893477    2544 patch_prober.go:28] interesting pod/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 11 21:56:15.893935 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:15.893530    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk" podUID="40e595e7-a303-4af1-b7ba-cd43a4407b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 11 21:56:16.893086 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:16.893051    2544 patch_prober.go:28] interesting pod/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 11 21:56:16.893287 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:16.893103    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk" podUID="40e595e7-a303-4af1-b7ba-cd43a4407b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 11 21:56:17.893057 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:17.893020    2544 patch_prober.go:28] interesting pod/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 11 21:56:17.893534 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:17.893089    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk" podUID="40e595e7-a303-4af1-b7ba-cd43a4407b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 11 21:56:18.897995 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:18.897958    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:56:18.898387 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:18.898259    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:56:18.898827 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:18.898813    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:56:19.829199 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:19.829164    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24efea-k5mkp"]
Sep 11 21:56:19.829556 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:19.829543    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291346f6-c663-433d-b065-6f3564dfd387" containerName="revoke-keys"
Sep 11 21:56:19.829603 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:19.829559    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="291346f6-c663-433d-b065-6f3564dfd387" containerName="revoke-keys"
Sep 11 21:56:19.829603 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:19.829578    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9311bc7-5cb2-4261-9084-43b77f0f8402" containerName="payload-processing"
Sep 11 21:56:19.829603 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:19.829584    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9311bc7-5cb2-4261-9084-43b77f0f8402" containerName="payload-processing"
Sep 11 21:56:19.829694 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:19.829656    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9311bc7-5cb2-4261-9084-43b77f0f8402" containerName="payload-processing"
Sep 11 21:56:19.829694 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:19.829664    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="291346f6-c663-433d-b065-6f3564dfd387" containerName="revoke-keys"
Sep 11 21:56:19.832439 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:19.832423    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24efea-k5mkp"
Sep 11 21:56:19.834763 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:19.834734    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 21:56:19.834904 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:19.834883    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:56:19.840465 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:19.840440    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24efea-k5mkp"]
Sep 11 21:56:19.993184 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:19.993142    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d775bd8-90ca-47bb-af06-115343d45d4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-24efea-k5mkp\" (UID: \"3d775bd8-90ca-47bb-af06-115343d45d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24efea-k5mkp"
Sep 11 21:56:20.094443 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:20.094351    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d775bd8-90ca-47bb-af06-115343d45d4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-24efea-k5mkp\" (UID: \"3d775bd8-90ca-47bb-af06-115343d45d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24efea-k5mkp"
Sep 11 21:56:20.095000 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:20.094977    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d775bd8-90ca-47bb-af06-115343d45d4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-24efea-k5mkp\" (UID: \"3d775bd8-90ca-47bb-af06-115343d45d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24efea-k5mkp"
Sep 11 21:56:20.144317 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:20.144279    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24efea-k5mkp"
Sep 11 21:56:20.414891 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:20.414851    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24efea-k5mkp"]
Sep 11 21:56:20.416558 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:56:20.416530    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d775bd8_90ca_47bb_af06_115343d45d4a.slice/crio-66044dc4ef2dc55dcef0db5cb5ff4932e575cc0b46ba13412a9acab2e285998d WatchSource:0}: Error finding container 66044dc4ef2dc55dcef0db5cb5ff4932e575cc0b46ba13412a9acab2e285998d: Status 404 returned error can't find the container with id 66044dc4ef2dc55dcef0db5cb5ff4932e575cc0b46ba13412a9acab2e285998d
Sep 11 21:56:21.038341 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:21.038250    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24efea-k5mkp" event={"ID":"3d775bd8-90ca-47bb-af06-115343d45d4a","Type":"ContainerStarted","Data":"68158084d6268feedd5de2e4e530b220a02d1fb3d126064271fda92bdedaf8b4"}
Sep 11 21:56:21.038341 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:21.038304    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24efea-k5mkp" event={"ID":"3d775bd8-90ca-47bb-af06-115343d45d4a","Type":"ContainerStarted","Data":"66044dc4ef2dc55dcef0db5cb5ff4932e575cc0b46ba13412a9acab2e285998d"}
Sep 11 21:56:21.054211 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:21.054155    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24efea-k5mkp" podStartSLOduration=2.054138686 podStartE2EDuration="2.054138686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:56:21.051818273 +0000 UTC m=+1544.021235095" watchObservedRunningTime="2026-09-11 21:56:21.054138686 +0000 UTC m=+1544.023555504"
Sep 11 21:56:22.044960 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:22.044924    2544 generic.go:356] "Generic (PLEG): container finished" podID="3d775bd8-90ca-47bb-af06-115343d45d4a" containerID="68158084d6268feedd5de2e4e530b220a02d1fb3d126064271fda92bdedaf8b4" exitCode=0
Sep 11 21:56:22.045443 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:22.045017    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24efea-k5mkp" event={"ID":"3d775bd8-90ca-47bb-af06-115343d45d4a","Type":"ContainerDied","Data":"68158084d6268feedd5de2e4e530b220a02d1fb3d126064271fda92bdedaf8b4"}
Sep 11 21:56:23.184920 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:23.184896    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24efea-k5mkp"
Sep 11 21:56:23.229550 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:23.229515    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d775bd8-90ca-47bb-af06-115343d45d4a-maas-api-service-ca\") pod \"3d775bd8-90ca-47bb-af06-115343d45d4a\" (UID: \"3d775bd8-90ca-47bb-af06-115343d45d4a\") "
Sep 11 21:56:23.229944 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:23.229917    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d775bd8-90ca-47bb-af06-115343d45d4a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d775bd8-90ca-47bb-af06-115343d45d4a" (UID: "3d775bd8-90ca-47bb-af06-115343d45d4a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:56:23.330737 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:23.330653    2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d775bd8-90ca-47bb-af06-115343d45d4a-maas-api-service-ca\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:24.056577 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:24.056545    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24efea-k5mkp"
Sep 11 21:56:24.056751 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:24.056552    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24efea-k5mkp" event={"ID":"3d775bd8-90ca-47bb-af06-115343d45d4a","Type":"ContainerDied","Data":"66044dc4ef2dc55dcef0db5cb5ff4932e575cc0b46ba13412a9acab2e285998d"}
Sep 11 21:56:24.056751 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:24.056665    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66044dc4ef2dc55dcef0db5cb5ff4932e575cc0b46ba13412a9acab2e285998d"
Sep 11 21:56:27.738167 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:27.738128    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f12741-drdx8"]
Sep 11 21:56:27.738569 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:27.738528    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d775bd8-90ca-47bb-af06-115343d45d4a" containerName="revoke-keys"
Sep 11 21:56:27.738569 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:27.738541    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d775bd8-90ca-47bb-af06-115343d45d4a" containerName="revoke-keys"
Sep 11 21:56:27.738641 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:27.738608    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d775bd8-90ca-47bb-af06-115343d45d4a" containerName="revoke-keys"
Sep 11 21:56:27.740862 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:27.740847    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f12741-drdx8"
Sep 11 21:56:27.743091 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:27.743067    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:56:27.743183 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:27.743147    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 21:56:27.748819 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:27.748797    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f12741-drdx8"]
Sep 11 21:56:27.763644 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:27.763620    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1563581f-9153-4cea-86c1-3fac95311992-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f12741-drdx8\" (UID: \"1563581f-9153-4cea-86c1-3fac95311992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f12741-drdx8"
Sep 11 21:56:27.864186 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:27.864155    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1563581f-9153-4cea-86c1-3fac95311992-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f12741-drdx8\" (UID: \"1563581f-9153-4cea-86c1-3fac95311992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f12741-drdx8"
Sep 11 21:56:27.864783 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:27.864764    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1563581f-9153-4cea-86c1-3fac95311992-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f12741-drdx8\" (UID: \"1563581f-9153-4cea-86c1-3fac95311992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f12741-drdx8"
Sep 11 21:56:28.052247 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:28.052136    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f12741-drdx8"
Sep 11 21:56:28.320011 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:28.319978    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f12741-drdx8"]
Sep 11 21:56:28.321642 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:56:28.321611    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1563581f_9153_4cea_86c1_3fac95311992.slice/crio-f7476d4e7bdd64017040175e5f91665b0691e062c36f924be3968bce586da01d WatchSource:0}: Error finding container f7476d4e7bdd64017040175e5f91665b0691e062c36f924be3968bce586da01d: Status 404 returned error can't find the container with id f7476d4e7bdd64017040175e5f91665b0691e062c36f924be3968bce586da01d
Sep 11 21:56:29.083865 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:29.083824    2544 generic.go:356] "Generic (PLEG): container finished" podID="1563581f-9153-4cea-86c1-3fac95311992" containerID="7aaab3341bf608d2932dace01aeac27457a2f4162435e8cfccb6302dcdfe4172" exitCode=0
Sep 11 21:56:29.084233 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:29.083902    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f12741-drdx8" event={"ID":"1563581f-9153-4cea-86c1-3fac95311992","Type":"ContainerDied","Data":"7aaab3341bf608d2932dace01aeac27457a2f4162435e8cfccb6302dcdfe4172"}
Sep 11 21:56:29.084233 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:29.083950    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f12741-drdx8" event={"ID":"1563581f-9153-4cea-86c1-3fac95311992","Type":"ContainerStarted","Data":"f7476d4e7bdd64017040175e5f91665b0691e062c36f924be3968bce586da01d"}
Sep 11 21:56:30.226122 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:30.226098    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f12741-drdx8"
Sep 11 21:56:30.286966 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:30.286933    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1563581f-9153-4cea-86c1-3fac95311992-maas-api-service-ca\") pod \"1563581f-9153-4cea-86c1-3fac95311992\" (UID: \"1563581f-9153-4cea-86c1-3fac95311992\") "
Sep 11 21:56:30.287353 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:30.287334    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1563581f-9153-4cea-86c1-3fac95311992-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1563581f-9153-4cea-86c1-3fac95311992" (UID: "1563581f-9153-4cea-86c1-3fac95311992"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:56:30.388778 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:30.388683    2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1563581f-9153-4cea-86c1-3fac95311992-maas-api-service-ca\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:31.095678 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:31.095648    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f12741-drdx8"
Sep 11 21:56:31.095837 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:31.095688    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f12741-drdx8" event={"ID":"1563581f-9153-4cea-86c1-3fac95311992","Type":"ContainerDied","Data":"f7476d4e7bdd64017040175e5f91665b0691e062c36f924be3968bce586da01d"}
Sep 11 21:56:31.095837 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:31.095734    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7476d4e7bdd64017040175e5f91665b0691e062c36f924be3968bce586da01d"
Sep 11 21:56:32.606178 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:32.606141    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"]
Sep 11 21:56:32.630677 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:32.630646    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"]
Sep 11 21:56:32.638782 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:32.638753    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"]
Sep 11 21:56:36.031986 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:36.031943    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24efea-k5mkp"]
Sep 11 21:56:36.034863 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:36.034834    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24efea-k5mkp"]
Sep 11 21:56:37.592228 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:37.592185    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d775bd8-90ca-47bb-af06-115343d45d4a" path="/var/lib/kubelet/pods/3d775bd8-90ca-47bb-af06-115343d45d4a/volumes"
Sep 11 21:56:37.811016 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:37.810986    2544 scope.go:117] "RemoveContainer" containerID="5e38f6db06d0b00d41bd3dddff924a226012b285704e877cf9840cd0932277a8"
Sep 11 21:56:42.584831 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:42.584802    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"]
Sep 11 21:56:42.585459 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:42.585003    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk" podUID="40e595e7-a303-4af1-b7ba-cd43a4407b99" containerName="istio-proxy" containerID="cri-o://6bab0200d462cc8844c469e373b5e5ba7776c817102d74b65ee7e0f34765d6f4" gracePeriod=30
Sep 11 21:56:42.934853 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:42.934775    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5cb5f4-n2j74"]
Sep 11 21:56:42.935145 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:42.935134    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1563581f-9153-4cea-86c1-3fac95311992" containerName="revoke-keys"
Sep 11 21:56:42.935188 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:42.935147    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="1563581f-9153-4cea-86c1-3fac95311992" containerName="revoke-keys"
Sep 11 21:56:42.935241 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:42.935210    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="1563581f-9153-4cea-86c1-3fac95311992" containerName="revoke-keys"
Sep 11 21:56:42.938337 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:42.938319    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5cb5f4-n2j74"
Sep 11 21:56:42.940825 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:42.940805    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:56:42.941033 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:42.941015    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 21:56:42.946186 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:42.946163    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5cb5f4-n2j74"]
Sep 11 21:56:43.006465 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:43.006437    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5cb5f4-n2j74\" (UID: \"c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5cb5f4-n2j74"
Sep 11 21:56:43.107807 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:43.107779    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5cb5f4-n2j74\" (UID: \"c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5cb5f4-n2j74"
Sep 11 21:56:43.108397 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:43.108378    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5cb5f4-n2j74\" (UID: \"c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5cb5f4-n2j74"
Sep 11 21:56:43.250807 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:43.250778    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5cb5f4-n2j74"
Sep 11 21:56:43.527280 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:43.527251    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5cb5f4-n2j74"]
Sep 11 21:56:43.529055 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:56:43.529031    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f4aae5_3e1d_4f61_bd05_46d9e4fb6075.slice/crio-15400fc2463f2586c4266e15f7816d50593f9a902d54d39dff7552de3d1610ba WatchSource:0}: Error finding container 15400fc2463f2586c4266e15f7816d50593f9a902d54d39dff7552de3d1610ba: Status 404 returned error can't find the container with id 15400fc2463f2586c4266e15f7816d50593f9a902d54d39dff7552de3d1610ba
Sep 11 21:56:44.168242 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:44.168200    2544 generic.go:356] "Generic (PLEG): container finished" podID="c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075" containerID="3b32b92f9adab9a46b970f1ea2996bdf042879a71c91bfc3ade0d0110629316a" exitCode=0
Sep 11 21:56:44.168512 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:44.168261    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5cb5f4-n2j74" event={"ID":"c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075","Type":"ContainerDied","Data":"3b32b92f9adab9a46b970f1ea2996bdf042879a71c91bfc3ade0d0110629316a"}
Sep 11 21:56:44.168512 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:44.168326    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5cb5f4-n2j74" event={"ID":"c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075","Type":"ContainerStarted","Data":"15400fc2463f2586c4266e15f7816d50593f9a902d54d39dff7552de3d1610ba"}
Sep 11 21:56:45.273612 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:45.273583    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f12741-29819385-6zjsp"]
Sep 11 21:56:45.276995 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:45.276967    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f12741-29819385-6zjsp"]
Sep 11 21:56:45.318197 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:45.318165    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5cb5f4-n2j74"
Sep 11 21:56:45.432078 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:45.431999    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075-maas-api-service-ca\") pod \"c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075\" (UID: \"c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075\") "
Sep 11 21:56:45.432387 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:45.432366    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075" (UID: "c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:56:45.525847 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:45.525817    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f12741-drdx8"]
Sep 11 21:56:45.532820 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:45.532794    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f12741-drdx8"]
Sep 11 21:56:45.533247 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:45.533207    2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075-maas-api-service-ca\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:45.592683 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:45.592655    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a025137-55cd-482f-82bf-63e9d4265b95" path="/var/lib/kubelet/pods/0a025137-55cd-482f-82bf-63e9d4265b95/volumes"
Sep 11 21:56:45.592981 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:45.592969    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1563581f-9153-4cea-86c1-3fac95311992" path="/var/lib/kubelet/pods/1563581f-9153-4cea-86c1-3fac95311992/volumes"
Sep 11 21:56:46.181144 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:46.181102    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5cb5f4-n2j74" event={"ID":"c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075","Type":"ContainerDied","Data":"15400fc2463f2586c4266e15f7816d50593f9a902d54d39dff7552de3d1610ba"}
Sep 11 21:56:46.181376 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:46.181143    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5cb5f4-n2j74"
Sep 11 21:56:46.181376 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:46.181168    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15400fc2463f2586c4266e15f7816d50593f9a902d54d39dff7552de3d1610ba"
Sep 11 21:56:47.624928 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:47.624892    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg" podUID="ede62c5f-0b90-49f9-a61d-e5e4209a1a65" containerName="main" containerID="cri-o://f1dca7ba74f817d1fb74ee1e1d5db952b9c9e53ed24cd78614abee42b40e12ba" gracePeriod=45
Sep 11 21:56:47.649541 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:47.649510    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm" podUID="62d25379-ff07-4c27-92a4-9e1a391c6e36" containerName="main" containerID="cri-o://e571aadad0e61ca42f9e6d1292aaf8484ef87e27b8d6c82fcb4d47d454ddbae0" gracePeriod=45
Sep 11 21:56:47.661566 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:47.661537    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq" podUID="56ec433b-c8ea-44a6-8f0b-1032239369f7" containerName="main" containerID="cri-o://8098522d20d28906e64aa9f19b7330b705a384e8d5cf6d07ffd1cc91aa27add9" gracePeriod=45
Sep 11 21:56:47.910675 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:47.910146    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:56:47.946165 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:47.946140    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:56:47.979316 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:47.979279    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:56:47.992119 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:47.992095    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:56:48.058642 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.058615    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-model-cache\") pod \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") "
Sep 11 21:56:48.058796 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.058650    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-tls-certs\") pod \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") "
Sep 11 21:56:48.058796 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.058690    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf6n9\" (UniqueName: \"kubernetes.io/projected/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-kube-api-access-pf6n9\") pod \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") "
Sep 11 21:56:48.058796 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.058718    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-home\") pod \"62d25379-ff07-4c27-92a4-9e1a391c6e36\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") "
Sep 11 21:56:48.058796 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.058746    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-647zp\" (UniqueName: \"kubernetes.io/projected/62d25379-ff07-4c27-92a4-9e1a391c6e36-kube-api-access-647zp\") pod \"62d25379-ff07-4c27-92a4-9e1a391c6e36\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") "
Sep 11 21:56:48.058796 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.058784    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-workload-socket\") pod \"40e595e7-a303-4af1-b7ba-cd43a4407b99\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") "
Sep 11 21:56:48.059040 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.058816    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-podinfo\") pod \"40e595e7-a303-4af1-b7ba-cd43a4407b99\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") "
Sep 11 21:56:48.059040 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.058842    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-home\") pod \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") "
Sep 11 21:56:48.059040 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.058876    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrslb\" (UniqueName: \"kubernetes.io/projected/56ec433b-c8ea-44a6-8f0b-1032239369f7-kube-api-access-qrslb\") pod \"56ec433b-c8ea-44a6-8f0b-1032239369f7\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") "
Sep 11 21:56:48.059040 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.058910    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-model-cache" (OuterVolumeSpecName: "model-cache") pod "ede62c5f-0b90-49f9-a61d-e5e4209a1a65" (UID: "ede62c5f-0b90-49f9-a61d-e5e4209a1a65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:48.059040 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.058919    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-tmp-dir\") pod \"56ec433b-c8ea-44a6-8f0b-1032239369f7\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") "
Sep 11 21:56:48.059040 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.058994    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-credential-socket\") pod \"40e595e7-a303-4af1-b7ba-cd43a4407b99\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") "
Sep 11 21:56:48.059040 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.058997    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-home" (OuterVolumeSpecName: "home") pod "62d25379-ff07-4c27-92a4-9e1a391c6e36" (UID: "62d25379-ff07-4c27-92a4-9e1a391c6e36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:48.059040 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.059029    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ec433b-c8ea-44a6-8f0b-1032239369f7-tls-certs\") pod \"56ec433b-c8ea-44a6-8f0b-1032239369f7\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") "
Sep 11 21:56:48.059445 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.059056    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-dshm\") pod \"56ec433b-c8ea-44a6-8f0b-1032239369f7\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") "
Sep 11 21:56:48.059445 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.059105    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d25379-ff07-4c27-92a4-9e1a391c6e36-tls-certs\") pod \"62d25379-ff07-4c27-92a4-9e1a391c6e36\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") "
Sep 11 21:56:48.059445 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.059116    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56ec433b-c8ea-44a6-8f0b-1032239369f7" (UID: "56ec433b-c8ea-44a6-8f0b-1032239369f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:48.059445 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.059132    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-model-cache\") pod \"62d25379-ff07-4c27-92a4-9e1a391c6e36\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") "
Sep 11 21:56:48.059445 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.059155    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-home" (OuterVolumeSpecName: "home") pod "ede62c5f-0b90-49f9-a61d-e5e4209a1a65" (UID: "ede62c5f-0b90-49f9-a61d-e5e4209a1a65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:48.059445 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.059238    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "40e595e7-a303-4af1-b7ba-cd43a4407b99" (UID: "40e595e7-a303-4af1-b7ba-cd43a4407b99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:48.059445 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.059337    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-model-cache" (OuterVolumeSpecName: "model-cache") pod "62d25379-ff07-4c27-92a4-9e1a391c6e36" (UID: "62d25379-ff07-4c27-92a4-9e1a391c6e36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:48.059729 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.059167    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "40e595e7-a303-4af1-b7ba-cd43a4407b99" (UID: "40e595e7-a303-4af1-b7ba-cd43a4407b99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:48.059825 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.059801    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e595e7-a303-4af1-b7ba-cd43a4407b99-istiod-ca-cert\") pod \"40e595e7-a303-4af1-b7ba-cd43a4407b99\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") "
Sep 11 21:56:48.059955 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.059895    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-workload-certs\") pod \"40e595e7-a303-4af1-b7ba-cd43a4407b99\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") "
Sep 11 21:56:48.060012 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.059958    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-home\") pod \"56ec433b-c8ea-44a6-8f0b-1032239369f7\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") "
Sep 11 21:56:48.060012 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.059996    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-envoy\") pod \"40e595e7-a303-4af1-b7ba-cd43a4407b99\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") "
Sep 11 21:56:48.060110 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.060032    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-tmp-dir\") pod \"62d25379-ff07-4c27-92a4-9e1a391c6e36\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") "
Sep 11 21:56:48.060110 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.060080    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nfd9\" (UniqueName: \"kubernetes.io/projected/40e595e7-a303-4af1-b7ba-cd43a4407b99-kube-api-access-8nfd9\") pod \"40e595e7-a303-4af1-b7ba-cd43a4407b99\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") "
Sep 11 21:56:48.060268 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.060127    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-dshm\") pod \"62d25379-ff07-4c27-92a4-9e1a391c6e36\" (UID: \"62d25379-ff07-4c27-92a4-9e1a391c6e36\") "
Sep 11 21:56:48.060268 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.060164    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-dshm\") pod \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") "
Sep 11 21:56:48.060268 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.060195    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-tmp-dir\") pod \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\" (UID: \"ede62c5f-0b90-49f9-a61d-e5e4209a1a65\") "
Sep 11 21:56:48.060268 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.060256    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-data\") pod \"40e595e7-a303-4af1-b7ba-cd43a4407b99\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") "
Sep 11 21:56:48.060473 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.060291    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-token\") pod \"40e595e7-a303-4af1-b7ba-cd43a4407b99\" (UID: \"40e595e7-a303-4af1-b7ba-cd43a4407b99\") "
Sep 11 21:56:48.060473 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.060317    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-model-cache\") pod \"56ec433b-c8ea-44a6-8f0b-1032239369f7\" (UID: \"56ec433b-c8ea-44a6-8f0b-1032239369f7\") "
Sep 11 21:56:48.060997 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.060938    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ede62c5f-0b90-49f9-a61d-e5e4209a1a65" (UID: "ede62c5f-0b90-49f9-a61d-e5e4209a1a65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:48.062273 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.062244    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "40e595e7-a303-4af1-b7ba-cd43a4407b99" (UID: "40e595e7-a303-4af1-b7ba-cd43a4407b99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:56:48.062758 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.062527    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-data" (OuterVolumeSpecName: "istio-data") pod "40e595e7-a303-4af1-b7ba-cd43a4407b99" (UID: "40e595e7-a303-4af1-b7ba-cd43a4407b99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:48.062758 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.062544    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-home" (OuterVolumeSpecName: "home") pod "56ec433b-c8ea-44a6-8f0b-1032239369f7" (UID: "56ec433b-c8ea-44a6-8f0b-1032239369f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:48.062920 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.062782    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "40e595e7-a303-4af1-b7ba-cd43a4407b99" (UID: "40e595e7-a303-4af1-b7ba-cd43a4407b99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:48.063175 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.062982    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62d25379-ff07-4c27-92a4-9e1a391c6e36" (UID: "62d25379-ff07-4c27-92a4-9e1a391c6e36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:48.063175 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.063038    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.063175 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.063048    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56ec433b-c8ea-44a6-8f0b-1032239369f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56ec433b-c8ea-44a6-8f0b-1032239369f7" (UID: "56ec433b-c8ea-44a6-8f0b-1032239369f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:56:48.063175 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.063057    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.063175 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.063091    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.063175 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.063116    2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-workload-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.063497 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.063204    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.063497 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.063233    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.063497 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.063247    2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-credential-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.063497 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.063266    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.063497 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.063480    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-dshm" (OuterVolumeSpecName: "dshm") pod "62d25379-ff07-4c27-92a4-9e1a391c6e36" (UID: "62d25379-ff07-4c27-92a4-9e1a391c6e36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:48.063959 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.063789    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "56ec433b-c8ea-44a6-8f0b-1032239369f7" (UID: "56ec433b-c8ea-44a6-8f0b-1032239369f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:48.064177 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.064129    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d25379-ff07-4c27-92a4-9e1a391c6e36-kube-api-access-647zp" (OuterVolumeSpecName: "kube-api-access-647zp") pod "62d25379-ff07-4c27-92a4-9e1a391c6e36" (UID: "62d25379-ff07-4c27-92a4-9e1a391c6e36"). InnerVolumeSpecName "kube-api-access-647zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:48.066322 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.064470    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-dshm" (OuterVolumeSpecName: "dshm") pod "ede62c5f-0b90-49f9-a61d-e5e4209a1a65" (UID: "ede62c5f-0b90-49f9-a61d-e5e4209a1a65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:48.066322 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.064477    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e595e7-a303-4af1-b7ba-cd43a4407b99-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "40e595e7-a303-4af1-b7ba-cd43a4407b99" (UID: "40e595e7-a303-4af1-b7ba-cd43a4407b99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:56:48.066322 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.064857    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e595e7-a303-4af1-b7ba-cd43a4407b99-kube-api-access-8nfd9" (OuterVolumeSpecName: "kube-api-access-8nfd9") pod "40e595e7-a303-4af1-b7ba-cd43a4407b99" (UID: "40e595e7-a303-4af1-b7ba-cd43a4407b99"). InnerVolumeSpecName "kube-api-access-8nfd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:48.067524 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.067496    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ede62c5f-0b90-49f9-a61d-e5e4209a1a65" (UID: "ede62c5f-0b90-49f9-a61d-e5e4209a1a65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:56:48.070173 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.070146    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d25379-ff07-4c27-92a4-9e1a391c6e36-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62d25379-ff07-4c27-92a4-9e1a391c6e36" (UID: "62d25379-ff07-4c27-92a4-9e1a391c6e36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:56:48.070300 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.070277    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-dshm" (OuterVolumeSpecName: "dshm") pod "56ec433b-c8ea-44a6-8f0b-1032239369f7" (UID: "56ec433b-c8ea-44a6-8f0b-1032239369f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:48.070369 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.070258    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ec433b-c8ea-44a6-8f0b-1032239369f7-kube-api-access-qrslb" (OuterVolumeSpecName: "kube-api-access-qrslb") pod "56ec433b-c8ea-44a6-8f0b-1032239369f7" (UID: "56ec433b-c8ea-44a6-8f0b-1032239369f7"). InnerVolumeSpecName "kube-api-access-qrslb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:48.070369 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.070333    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-kube-api-access-pf6n9" (OuterVolumeSpecName: "kube-api-access-pf6n9") pod "ede62c5f-0b90-49f9-a61d-e5e4209a1a65" (UID: "ede62c5f-0b90-49f9-a61d-e5e4209a1a65"). InnerVolumeSpecName "kube-api-access-pf6n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:48.070878 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.070856    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-token" (OuterVolumeSpecName: "istio-token") pod "40e595e7-a303-4af1-b7ba-cd43a4407b99" (UID: "40e595e7-a303-4af1-b7ba-cd43a4407b99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:48.071147 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.071123    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "40e595e7-a303-4af1-b7ba-cd43a4407b99" (UID: "40e595e7-a303-4af1-b7ba-cd43a4407b99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:48.164020 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.163944    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d25379-ff07-4c27-92a4-9e1a391c6e36-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.164020 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.163973    2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e595e7-a303-4af1-b7ba-cd43a4407b99-istiod-ca-cert\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.164020 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.163982    2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-workload-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.164020 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.163991    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.164020 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.163998    2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-envoy\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.164020 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.164005    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.164020 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.164013    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nfd9\" (UniqueName: \"kubernetes.io/projected/40e595e7-a303-4af1-b7ba-cd43a4407b99-kube-api-access-8nfd9\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.164020 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.164022    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62d25379-ff07-4c27-92a4-9e1a391c6e36-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.164377 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.164030    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.164377 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.164037    2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-data\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.164377 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.164046    2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-token\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.164377 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.164053    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.164377 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.164060    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.164377 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.164068    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf6n9\" (UniqueName: \"kubernetes.io/projected/ede62c5f-0b90-49f9-a61d-e5e4209a1a65-kube-api-access-pf6n9\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.164377 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.164075    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-647zp\" (UniqueName: \"kubernetes.io/projected/62d25379-ff07-4c27-92a4-9e1a391c6e36-kube-api-access-647zp\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.164377 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.164083    2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e595e7-a303-4af1-b7ba-cd43a4407b99-istio-podinfo\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.164377 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.164090    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrslb\" (UniqueName: \"kubernetes.io/projected/56ec433b-c8ea-44a6-8f0b-1032239369f7-kube-api-access-qrslb\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.164377 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.164098    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ec433b-c8ea-44a6-8f0b-1032239369f7-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.164377 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.164105    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ec433b-c8ea-44a6-8f0b-1032239369f7-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:48.195565 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.195528    2544 generic.go:356] "Generic (PLEG): container finished" podID="40e595e7-a303-4af1-b7ba-cd43a4407b99" containerID="6bab0200d462cc8844c469e373b5e5ba7776c817102d74b65ee7e0f34765d6f4" exitCode=0
Sep 11 21:56:48.195717 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.195593    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"
Sep 11 21:56:48.195717 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.195608    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk" event={"ID":"40e595e7-a303-4af1-b7ba-cd43a4407b99","Type":"ContainerDied","Data":"6bab0200d462cc8844c469e373b5e5ba7776c817102d74b65ee7e0f34765d6f4"}
Sep 11 21:56:48.195717 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.195696    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk" event={"ID":"40e595e7-a303-4af1-b7ba-cd43a4407b99","Type":"ContainerDied","Data":"0b07f8c29709a261b4e6a6f80231462596a9bf927efd0422642b2e63d37a7219"}
Sep 11 21:56:48.195891 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.195726    2544 scope.go:117] "RemoveContainer" containerID="6bab0200d462cc8844c469e373b5e5ba7776c817102d74b65ee7e0f34765d6f4"
Sep 11 21:56:48.197576 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.197556    2544 generic.go:356] "Generic (PLEG): container finished" podID="56ec433b-c8ea-44a6-8f0b-1032239369f7" containerID="8098522d20d28906e64aa9f19b7330b705a384e8d5cf6d07ffd1cc91aa27add9" exitCode=0
Sep 11 21:56:48.197693 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.197630    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"
Sep 11 21:56:48.197693 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.197651    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq" event={"ID":"56ec433b-c8ea-44a6-8f0b-1032239369f7","Type":"ContainerDied","Data":"8098522d20d28906e64aa9f19b7330b705a384e8d5cf6d07ffd1cc91aa27add9"}
Sep 11 21:56:48.197793 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.197692    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq" event={"ID":"56ec433b-c8ea-44a6-8f0b-1032239369f7","Type":"ContainerDied","Data":"ebacbb1a09ee389c17d9a4603fc7e5daab8993fdefe3678ee181f30f124e0f23"}
Sep 11 21:56:48.199118 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.199088    2544 generic.go:356] "Generic (PLEG): container finished" podID="ede62c5f-0b90-49f9-a61d-e5e4209a1a65" containerID="f1dca7ba74f817d1fb74ee1e1d5db952b9c9e53ed24cd78614abee42b40e12ba" exitCode=0
Sep 11 21:56:48.199266 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.199135    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg" event={"ID":"ede62c5f-0b90-49f9-a61d-e5e4209a1a65","Type":"ContainerDied","Data":"f1dca7ba74f817d1fb74ee1e1d5db952b9c9e53ed24cd78614abee42b40e12ba"}
Sep 11 21:56:48.199266 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.199153    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"
Sep 11 21:56:48.199266 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.199165    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg" event={"ID":"ede62c5f-0b90-49f9-a61d-e5e4209a1a65","Type":"ContainerDied","Data":"011fd26c4760795328fe427216d8c77ae2a9e5f2eaae036d49922ba3cad4a4a0"}
Sep 11 21:56:48.200812 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.200640    2544 generic.go:356] "Generic (PLEG): container finished" podID="62d25379-ff07-4c27-92a4-9e1a391c6e36" containerID="e571aadad0e61ca42f9e6d1292aaf8484ef87e27b8d6c82fcb4d47d454ddbae0" exitCode=0
Sep 11 21:56:48.200812 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.200669    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm" event={"ID":"62d25379-ff07-4c27-92a4-9e1a391c6e36","Type":"ContainerDied","Data":"e571aadad0e61ca42f9e6d1292aaf8484ef87e27b8d6c82fcb4d47d454ddbae0"}
Sep 11 21:56:48.200812 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.200700    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm" event={"ID":"62d25379-ff07-4c27-92a4-9e1a391c6e36","Type":"ContainerDied","Data":"f15c5f1bdac1a141dec004b4944d837093947dfa55afa3742e92323985bb9687"}
Sep 11 21:56:48.200812 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.200703    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"
Sep 11 21:56:48.206483 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.206452    2544 scope.go:117] "RemoveContainer" containerID="3623ffa890d60d48603a107cad9f214f0274966c7f3a98dd4fc56bb01ff39a4a"
Sep 11 21:56:48.215996 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.215979    2544 scope.go:117] "RemoveContainer" containerID="6bab0200d462cc8844c469e373b5e5ba7776c817102d74b65ee7e0f34765d6f4"
Sep 11 21:56:48.216272 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:56:48.216254    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bab0200d462cc8844c469e373b5e5ba7776c817102d74b65ee7e0f34765d6f4\": container with ID starting with 6bab0200d462cc8844c469e373b5e5ba7776c817102d74b65ee7e0f34765d6f4 not found: ID does not exist" containerID="6bab0200d462cc8844c469e373b5e5ba7776c817102d74b65ee7e0f34765d6f4"
Sep 11 21:56:48.216316 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.216278    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bab0200d462cc8844c469e373b5e5ba7776c817102d74b65ee7e0f34765d6f4"} err="failed to get container status \"6bab0200d462cc8844c469e373b5e5ba7776c817102d74b65ee7e0f34765d6f4\": rpc error: code = NotFound desc = could not find container \"6bab0200d462cc8844c469e373b5e5ba7776c817102d74b65ee7e0f34765d6f4\": container with ID starting with 6bab0200d462cc8844c469e373b5e5ba7776c817102d74b65ee7e0f34765d6f4 not found: ID does not exist"
Sep 11 21:56:48.216316 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.216293    2544 scope.go:117] "RemoveContainer" containerID="3623ffa890d60d48603a107cad9f214f0274966c7f3a98dd4fc56bb01ff39a4a"
Sep 11 21:56:48.216526 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:56:48.216508    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3623ffa890d60d48603a107cad9f214f0274966c7f3a98dd4fc56bb01ff39a4a\": container with ID starting with 3623ffa890d60d48603a107cad9f214f0274966c7f3a98dd4fc56bb01ff39a4a not found: ID does not exist" containerID="3623ffa890d60d48603a107cad9f214f0274966c7f3a98dd4fc56bb01ff39a4a"
Sep 11 21:56:48.216572 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.216532    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3623ffa890d60d48603a107cad9f214f0274966c7f3a98dd4fc56bb01ff39a4a"} err="failed to get container status \"3623ffa890d60d48603a107cad9f214f0274966c7f3a98dd4fc56bb01ff39a4a\": rpc error: code = NotFound desc = could not find container \"3623ffa890d60d48603a107cad9f214f0274966c7f3a98dd4fc56bb01ff39a4a\": container with ID starting with 3623ffa890d60d48603a107cad9f214f0274966c7f3a98dd4fc56bb01ff39a4a not found: ID does not exist"
Sep 11 21:56:48.216572 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.216546    2544 scope.go:117] "RemoveContainer" containerID="8098522d20d28906e64aa9f19b7330b705a384e8d5cf6d07ffd1cc91aa27add9"
Sep 11 21:56:48.225906 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.224966    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"]
Sep 11 21:56:48.225906 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.225022    2544 scope.go:117] "RemoveContainer" containerID="8098522d20d28906e64aa9f19b7330b705a384e8d5cf6d07ffd1cc91aa27add9"
Sep 11 21:56:48.225906 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:56:48.225421    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8098522d20d28906e64aa9f19b7330b705a384e8d5cf6d07ffd1cc91aa27add9\": container with ID starting with 8098522d20d28906e64aa9f19b7330b705a384e8d5cf6d07ffd1cc91aa27add9 not found: ID does not exist" containerID="8098522d20d28906e64aa9f19b7330b705a384e8d5cf6d07ffd1cc91aa27add9"
Sep 11 21:56:48.225906 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.225454    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8098522d20d28906e64aa9f19b7330b705a384e8d5cf6d07ffd1cc91aa27add9"} err="failed to get container status \"8098522d20d28906e64aa9f19b7330b705a384e8d5cf6d07ffd1cc91aa27add9\": rpc error: code = NotFound desc = could not find container \"8098522d20d28906e64aa9f19b7330b705a384e8d5cf6d07ffd1cc91aa27add9\": container with ID starting with 8098522d20d28906e64aa9f19b7330b705a384e8d5cf6d07ffd1cc91aa27add9 not found: ID does not exist"
Sep 11 21:56:48.225906 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.225474    2544 scope.go:117] "RemoveContainer" containerID="f1dca7ba74f817d1fb74ee1e1d5db952b9c9e53ed24cd78614abee42b40e12ba"
Sep 11 21:56:48.228949 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.228932    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-24efea-openshift-default-649dd7b56-rc7wk"]
Sep 11 21:56:48.233979 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.233965    2544 scope.go:117] "RemoveContainer" containerID="f1dca7ba74f817d1fb74ee1e1d5db952b9c9e53ed24cd78614abee42b40e12ba"
Sep 11 21:56:48.234244 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:56:48.234212    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1dca7ba74f817d1fb74ee1e1d5db952b9c9e53ed24cd78614abee42b40e12ba\": container with ID starting with f1dca7ba74f817d1fb74ee1e1d5db952b9c9e53ed24cd78614abee42b40e12ba not found: ID does not exist" containerID="f1dca7ba74f817d1fb74ee1e1d5db952b9c9e53ed24cd78614abee42b40e12ba"
Sep 11 21:56:48.234291 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.234250    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1dca7ba74f817d1fb74ee1e1d5db952b9c9e53ed24cd78614abee42b40e12ba"} err="failed to get container status \"f1dca7ba74f817d1fb74ee1e1d5db952b9c9e53ed24cd78614abee42b40e12ba\": rpc error: code = NotFound desc = could not find container \"f1dca7ba74f817d1fb74ee1e1d5db952b9c9e53ed24cd78614abee42b40e12ba\": container with ID starting with f1dca7ba74f817d1fb74ee1e1d5db952b9c9e53ed24cd78614abee42b40e12ba not found: ID does not exist"
Sep 11 21:56:48.234291 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.234264    2544 scope.go:117] "RemoveContainer" containerID="e571aadad0e61ca42f9e6d1292aaf8484ef87e27b8d6c82fcb4d47d454ddbae0"
Sep 11 21:56:48.240779 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.240755    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"]
Sep 11 21:56:48.241396 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.241380    2544 scope.go:117] "RemoveContainer" containerID="e571aadad0e61ca42f9e6d1292aaf8484ef87e27b8d6c82fcb4d47d454ddbae0"
Sep 11 21:56:48.241645 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:56:48.241627    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e571aadad0e61ca42f9e6d1292aaf8484ef87e27b8d6c82fcb4d47d454ddbae0\": container with ID starting with e571aadad0e61ca42f9e6d1292aaf8484ef87e27b8d6c82fcb4d47d454ddbae0 not found: ID does not exist" containerID="e571aadad0e61ca42f9e6d1292aaf8484ef87e27b8d6c82fcb4d47d454ddbae0"
Sep 11 21:56:48.241741 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.241653    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e571aadad0e61ca42f9e6d1292aaf8484ef87e27b8d6c82fcb4d47d454ddbae0"} err="failed to get container status \"e571aadad0e61ca42f9e6d1292aaf8484ef87e27b8d6c82fcb4d47d454ddbae0\": rpc error: code = NotFound desc = could not find container \"e571aadad0e61ca42f9e6d1292aaf8484ef87e27b8d6c82fcb4d47d454ddbae0\": container with ID starting with e571aadad0e61ca42f9e6d1292aaf8484ef87e27b8d6c82fcb4d47d454ddbae0 not found: ID does not exist"
Sep 11 21:56:48.244021 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.244002    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f12741/simulator-w0-f12741-kserve-f9895bb74-j57kq"]
Sep 11 21:56:48.253204 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.253184    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"]
Sep 11 21:56:48.256921 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.256901    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f12741/distinct-2-w0-f12741-kserve-5d7877d57-jv8jg"]
Sep 11 21:56:48.266624 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.266602    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"]
Sep 11 21:56:48.274765 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:48.274739    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f12741/premium-w0-f12741-kserve-c57b5578c-gjhnm"]
Sep 11 21:56:49.592125 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:49.592098    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40e595e7-a303-4af1-b7ba-cd43a4407b99" path="/var/lib/kubelet/pods/40e595e7-a303-4af1-b7ba-cd43a4407b99/volumes"
Sep 11 21:56:49.592507 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:49.592492    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56ec433b-c8ea-44a6-8f0b-1032239369f7" path="/var/lib/kubelet/pods/56ec433b-c8ea-44a6-8f0b-1032239369f7/volumes"
Sep 11 21:56:49.592840 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:49.592828    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62d25379-ff07-4c27-92a4-9e1a391c6e36" path="/var/lib/kubelet/pods/62d25379-ff07-4c27-92a4-9e1a391c6e36/volumes"
Sep 11 21:56:49.593508 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:49.593493    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ede62c5f-0b90-49f9-a61d-e5e4209a1a65" path="/var/lib/kubelet/pods/ede62c5f-0b90-49f9-a61d-e5e4209a1a65/volumes"
Sep 11 21:56:50.787285 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.787248    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-35e777-rvsbm"]
Sep 11 21:56:50.787646 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.787591    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d25379-ff07-4c27-92a4-9e1a391c6e36" containerName="main"
Sep 11 21:56:50.787646 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.787602    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d25379-ff07-4c27-92a4-9e1a391c6e36" containerName="main"
Sep 11 21:56:50.787646 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.787620    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ede62c5f-0b90-49f9-a61d-e5e4209a1a65" containerName="main"
Sep 11 21:56:50.787646 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.787625    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="ede62c5f-0b90-49f9-a61d-e5e4209a1a65" containerName="main"
Sep 11 21:56:50.787646 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.787631    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e595e7-a303-4af1-b7ba-cd43a4407b99" containerName="istio-proxy"
Sep 11 21:56:50.787646 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.787637    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e595e7-a303-4af1-b7ba-cd43a4407b99" containerName="istio-proxy"
Sep 11 21:56:50.787646 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.787648    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56ec433b-c8ea-44a6-8f0b-1032239369f7" containerName="main"
Sep 11 21:56:50.787898 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.787654    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="56ec433b-c8ea-44a6-8f0b-1032239369f7" containerName="main"
Sep 11 21:56:50.787898 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.787659    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075" containerName="revoke-keys"
Sep 11 21:56:50.787898 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.787666    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075" containerName="revoke-keys"
Sep 11 21:56:50.787898 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.787716    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="40e595e7-a303-4af1-b7ba-cd43a4407b99" containerName="istio-proxy"
Sep 11 21:56:50.787898 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.787724    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075" containerName="revoke-keys"
Sep 11 21:56:50.787898 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.787732    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d25379-ff07-4c27-92a4-9e1a391c6e36" containerName="main"
Sep 11 21:56:50.787898 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.787739    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="56ec433b-c8ea-44a6-8f0b-1032239369f7" containerName="main"
Sep 11 21:56:50.787898 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.787743    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="ede62c5f-0b90-49f9-a61d-e5e4209a1a65" containerName="main"
Sep 11 21:56:50.791872 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.791855    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-35e777-rvsbm"
Sep 11 21:56:50.794442 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.794422    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 21:56:50.794616 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.794490    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:56:50.798068 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.798044    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-35e777-rvsbm"]
Sep 11 21:56:50.890342 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.890313    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44d7aba-a895-4292-bbb9-ee87bc6952fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-35e777-rvsbm\" (UID: \"b44d7aba-a895-4292-bbb9-ee87bc6952fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-35e777-rvsbm"
Sep 11 21:56:50.990814 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.990778    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44d7aba-a895-4292-bbb9-ee87bc6952fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-35e777-rvsbm\" (UID: \"b44d7aba-a895-4292-bbb9-ee87bc6952fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-35e777-rvsbm"
Sep 11 21:56:50.991398 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:50.991380    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44d7aba-a895-4292-bbb9-ee87bc6952fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-35e777-rvsbm\" (UID: \"b44d7aba-a895-4292-bbb9-ee87bc6952fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-35e777-rvsbm"
Sep 11 21:56:51.102664 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:51.102581    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-35e777-rvsbm"
Sep 11 21:56:51.373332 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:51.373306    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-35e777-rvsbm"]
Sep 11 21:56:51.375254 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:56:51.375209    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44d7aba_a895_4292_bbb9_ee87bc6952fb.slice/crio-573c0fff98062ba97ac32b1104cb5e5ce4a4295c80f75c67ac360b042eee58fd WatchSource:0}: Error finding container 573c0fff98062ba97ac32b1104cb5e5ce4a4295c80f75c67ac360b042eee58fd: Status 404 returned error can't find the container with id 573c0fff98062ba97ac32b1104cb5e5ce4a4295c80f75c67ac360b042eee58fd
Sep 11 21:56:52.231628 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:52.231597    2544 generic.go:356] "Generic (PLEG): container finished" podID="b44d7aba-a895-4292-bbb9-ee87bc6952fb" containerID="211f8c4809bdc39ac2e5744e79e47d84de580dc590ffeb73f86e54e420660683" exitCode=0
Sep 11 21:56:52.231998 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:52.231670    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-35e777-rvsbm" event={"ID":"b44d7aba-a895-4292-bbb9-ee87bc6952fb","Type":"ContainerDied","Data":"211f8c4809bdc39ac2e5744e79e47d84de580dc590ffeb73f86e54e420660683"}
Sep 11 21:56:52.231998 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:52.231731    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-35e777-rvsbm" event={"ID":"b44d7aba-a895-4292-bbb9-ee87bc6952fb","Type":"ContainerStarted","Data":"573c0fff98062ba97ac32b1104cb5e5ce4a4295c80f75c67ac360b042eee58fd"}
Sep 11 21:56:53.366683 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:53.366658    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-35e777-rvsbm"
Sep 11 21:56:53.516139 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:53.516042    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44d7aba-a895-4292-bbb9-ee87bc6952fb-maas-api-service-ca\") pod \"b44d7aba-a895-4292-bbb9-ee87bc6952fb\" (UID: \"b44d7aba-a895-4292-bbb9-ee87bc6952fb\") "
Sep 11 21:56:53.516476 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:53.516456    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44d7aba-a895-4292-bbb9-ee87bc6952fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b44d7aba-a895-4292-bbb9-ee87bc6952fb" (UID: "b44d7aba-a895-4292-bbb9-ee87bc6952fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:56:53.617040 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:53.617012    2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44d7aba-a895-4292-bbb9-ee87bc6952fb-maas-api-service-ca\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:56:54.243765 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:54.243738    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-35e777-rvsbm"
Sep 11 21:56:54.243765 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:54.243752    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-35e777-rvsbm" event={"ID":"b44d7aba-a895-4292-bbb9-ee87bc6952fb","Type":"ContainerDied","Data":"573c0fff98062ba97ac32b1104cb5e5ce4a4295c80f75c67ac360b042eee58fd"}
Sep 11 21:56:54.243975 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:54.243796    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="573c0fff98062ba97ac32b1104cb5e5ce4a4295c80f75c67ac360b042eee58fd"
Sep 11 21:56:54.356542 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:54.356507    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5cb5f4-n2j74"]
Sep 11 21:56:54.358854 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:54.358825    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5cb5f4-n2j74"]
Sep 11 21:56:55.591950 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:55.591916    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075" path="/var/lib/kubelet/pods/c7f4aae5-3e1d-4f61-bd05-46d9e4fb6075/volumes"
Sep 11 21:56:55.656595 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:55.656565    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"]
Sep 11 21:56:55.661406 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:55.661383    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"]
Sep 11 21:56:55.671019 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:55.670995    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"]
Sep 11 21:56:55.682348 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:56:55.682323    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"]
Sep 11 21:57:09.539805 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:09.539770    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-35e777-29819385-jn7qd"]
Sep 11 21:57:09.542933 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:09.542903    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-35e777-29819385-jn7qd"]
Sep 11 21:57:09.594633 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:09.594597    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0627f81-4b5c-49f5-8315-83a9630667a9" path="/var/lib/kubelet/pods/c0627f81-4b5c-49f5-8315-83a9630667a9/volumes"
Sep 11 21:57:09.784300 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:09.784269    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-35e777-rvsbm"]
Sep 11 21:57:09.786094 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:09.786072    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-35e777-rvsbm"]
Sep 11 21:57:10.675613 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:10.675570    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r" podUID="17a22aae-0368-455b-9d83-308bc9760f26" containerName="main" containerID="cri-o://da62f719acf07490586bbd29ecb1e1b87190ad21f739445c7e8d709227633647" gracePeriod=45
Sep 11 21:57:10.679147 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:10.679113    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7" podUID="bca69aa4-8bb0-43b1-9113-82ad49cd5f6e" containerName="main" containerID="cri-o://d01f2d4ad6e599b2d69d67b0b7753165f5f1c6263581c1e555bb60376fed5485" gracePeriod=45
Sep 11 21:57:10.688619 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:10.688574    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2" podUID="f8ea762c-f334-4f8e-baa2-03a183358369" containerName="main" containerID="cri-o://6ebd0e3ec611f81bcceac85ab8bce12b198f0708813eff886e9d93956428603e" gracePeriod=45
Sep 11 21:57:10.706574 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:10.706531    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs" podUID="508b6f76-c334-4096-9d26-1c4000534efe" containerName="main" containerID="cri-o://5e530a8b0e9b7ff2c586ff6b36960de98928e8d8e75d89ee22252943fee82d3e" gracePeriod=45
Sep 11 21:57:10.992092 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:10.992067    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:57:11.008773 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.008724    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:57:11.022248 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.022192    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:57:11.028198 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.028174    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:57:11.088095 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088067    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-home\") pod \"508b6f76-c334-4096-9d26-1c4000534efe\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") "
Sep 11 21:57:11.088271 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088111    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508b6f76-c334-4096-9d26-1c4000534efe-tls-certs\") pod \"508b6f76-c334-4096-9d26-1c4000534efe\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") "
Sep 11 21:57:11.088271 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088142    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-tmp-dir\") pod \"f8ea762c-f334-4f8e-baa2-03a183358369\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") "
Sep 11 21:57:11.088271 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088156    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17a22aae-0368-455b-9d83-308bc9760f26-tls-certs\") pod \"17a22aae-0368-455b-9d83-308bc9760f26\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") "
Sep 11 21:57:11.088271 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088184    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-dshm\") pod \"17a22aae-0368-455b-9d83-308bc9760f26\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") "
Sep 11 21:57:11.088271 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088207    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-home\") pod \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") "
Sep 11 21:57:11.088271 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088260    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-home\") pod \"17a22aae-0368-455b-9d83-308bc9760f26\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") "
Sep 11 21:57:11.088592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088300    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ea762c-f334-4f8e-baa2-03a183358369-tls-certs\") pod \"f8ea762c-f334-4f8e-baa2-03a183358369\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") "
Sep 11 21:57:11.088592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088326    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5qmj\" (UniqueName: \"kubernetes.io/projected/508b6f76-c334-4096-9d26-1c4000534efe-kube-api-access-l5qmj\") pod \"508b6f76-c334-4096-9d26-1c4000534efe\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") "
Sep 11 21:57:11.088592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088350    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-tls-certs\") pod \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") "
Sep 11 21:57:11.088592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088397    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68hf2\" (UniqueName: \"kubernetes.io/projected/17a22aae-0368-455b-9d83-308bc9760f26-kube-api-access-68hf2\") pod \"17a22aae-0368-455b-9d83-308bc9760f26\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") "
Sep 11 21:57:11.088592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088426    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-dshm\") pod \"508b6f76-c334-4096-9d26-1c4000534efe\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") "
Sep 11 21:57:11.088592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088448    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-tmp-dir\") pod \"508b6f76-c334-4096-9d26-1c4000534efe\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") "
Sep 11 21:57:11.088592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088478    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-home\") pod \"f8ea762c-f334-4f8e-baa2-03a183358369\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") "
Sep 11 21:57:11.088592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088515    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8h5v\" (UniqueName: \"kubernetes.io/projected/f8ea762c-f334-4f8e-baa2-03a183358369-kube-api-access-j8h5v\") pod \"f8ea762c-f334-4f8e-baa2-03a183358369\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") "
Sep 11 21:57:11.088592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088390    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-home" (OuterVolumeSpecName: "home") pod "508b6f76-c334-4096-9d26-1c4000534efe" (UID: "508b6f76-c334-4096-9d26-1c4000534efe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:11.088592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088557    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftwp4\" (UniqueName: \"kubernetes.io/projected/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-kube-api-access-ftwp4\") pod \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") "
Sep 11 21:57:11.088592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088426    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f8ea762c-f334-4f8e-baa2-03a183358369" (UID: "f8ea762c-f334-4f8e-baa2-03a183358369"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:11.088592 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088590    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-model-cache\") pod \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") "
Sep 11 21:57:11.089174 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088603    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-home" (OuterVolumeSpecName: "home") pod "bca69aa4-8bb0-43b1-9113-82ad49cd5f6e" (UID: "bca69aa4-8bb0-43b1-9113-82ad49cd5f6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:11.089174 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088617    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-dshm\") pod \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") "
Sep 11 21:57:11.089174 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088610    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-home" (OuterVolumeSpecName: "home") pod "17a22aae-0368-455b-9d83-308bc9760f26" (UID: "17a22aae-0368-455b-9d83-308bc9760f26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:11.089174 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088644    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-dshm\") pod \"f8ea762c-f334-4f8e-baa2-03a183358369\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") "
Sep 11 21:57:11.089174 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088674    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-model-cache\") pod \"508b6f76-c334-4096-9d26-1c4000534efe\" (UID: \"508b6f76-c334-4096-9d26-1c4000534efe\") "
Sep 11 21:57:11.089174 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.088786    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-home" (OuterVolumeSpecName: "home") pod "f8ea762c-f334-4f8e-baa2-03a183358369" (UID: "f8ea762c-f334-4f8e-baa2-03a183358369"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:11.089174 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.089113    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-model-cache" (OuterVolumeSpecName: "model-cache") pod "bca69aa4-8bb0-43b1-9113-82ad49cd5f6e" (UID: "bca69aa4-8bb0-43b1-9113-82ad49cd5f6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:11.089619 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.089270    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-tmp-dir\") pod \"17a22aae-0368-455b-9d83-308bc9760f26\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") "
Sep 11 21:57:11.089619 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.089312    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-tmp-dir\") pod \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\" (UID: \"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e\") "
Sep 11 21:57:11.089619 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.089340    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-model-cache\") pod \"f8ea762c-f334-4f8e-baa2-03a183358369\" (UID: \"f8ea762c-f334-4f8e-baa2-03a183358369\") "
Sep 11 21:57:11.089619 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.089390    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-model-cache\") pod \"17a22aae-0368-455b-9d83-308bc9760f26\" (UID: \"17a22aae-0368-455b-9d83-308bc9760f26\") "
Sep 11 21:57:11.089814 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.089764    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.089814 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.089784    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.089814 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.089798    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.089814 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.089810    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.090018 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.089825    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.090018 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.089865    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.090144 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.090123    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-model-cache" (OuterVolumeSpecName: "model-cache") pod "17a22aae-0368-455b-9d83-308bc9760f26" (UID: "17a22aae-0368-455b-9d83-308bc9760f26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:11.090948 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.090644    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "508b6f76-c334-4096-9d26-1c4000534efe" (UID: "508b6f76-c334-4096-9d26-1c4000534efe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:11.091079 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.091065    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-model-cache" (OuterVolumeSpecName: "model-cache") pod "508b6f76-c334-4096-9d26-1c4000534efe" (UID: "508b6f76-c334-4096-9d26-1c4000534efe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:11.091251 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.091196    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bca69aa4-8bb0-43b1-9113-82ad49cd5f6e" (UID: "bca69aa4-8bb0-43b1-9113-82ad49cd5f6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:11.091333 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.091282    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-model-cache" (OuterVolumeSpecName: "model-cache") pod "f8ea762c-f334-4f8e-baa2-03a183358369" (UID: "f8ea762c-f334-4f8e-baa2-03a183358369"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:11.091700 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.091665    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17a22aae-0368-455b-9d83-308bc9760f26" (UID: "17a22aae-0368-455b-9d83-308bc9760f26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:11.093691 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.093668    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-dshm" (OuterVolumeSpecName: "dshm") pod "f8ea762c-f334-4f8e-baa2-03a183358369" (UID: "f8ea762c-f334-4f8e-baa2-03a183358369"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:11.093850 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.093821    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a22aae-0368-455b-9d83-308bc9760f26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17a22aae-0368-455b-9d83-308bc9760f26" (UID: "17a22aae-0368-455b-9d83-308bc9760f26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:57:11.094065 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.093940    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/508b6f76-c334-4096-9d26-1c4000534efe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "508b6f76-c334-4096-9d26-1c4000534efe" (UID: "508b6f76-c334-4096-9d26-1c4000534efe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:57:11.094065 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.093948    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-dshm" (OuterVolumeSpecName: "dshm") pod "bca69aa4-8bb0-43b1-9113-82ad49cd5f6e" (UID: "bca69aa4-8bb0-43b1-9113-82ad49cd5f6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:11.094065 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.093969    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a22aae-0368-455b-9d83-308bc9760f26-kube-api-access-68hf2" (OuterVolumeSpecName: "kube-api-access-68hf2") pod "17a22aae-0368-455b-9d83-308bc9760f26" (UID: "17a22aae-0368-455b-9d83-308bc9760f26"). InnerVolumeSpecName "kube-api-access-68hf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:57:11.094065 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.093996    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-kube-api-access-ftwp4" (OuterVolumeSpecName: "kube-api-access-ftwp4") pod "bca69aa4-8bb0-43b1-9113-82ad49cd5f6e" (UID: "bca69aa4-8bb0-43b1-9113-82ad49cd5f6e"). InnerVolumeSpecName "kube-api-access-ftwp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:57:11.094065 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.094002    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-dshm" (OuterVolumeSpecName: "dshm") pod "17a22aae-0368-455b-9d83-308bc9760f26" (UID: "17a22aae-0368-455b-9d83-308bc9760f26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:11.094065 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.094006    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ea762c-f334-4f8e-baa2-03a183358369-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8ea762c-f334-4f8e-baa2-03a183358369" (UID: "f8ea762c-f334-4f8e-baa2-03a183358369"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:57:11.094065 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.094034    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ea762c-f334-4f8e-baa2-03a183358369-kube-api-access-j8h5v" (OuterVolumeSpecName: "kube-api-access-j8h5v") pod "f8ea762c-f334-4f8e-baa2-03a183358369" (UID: "f8ea762c-f334-4f8e-baa2-03a183358369"). InnerVolumeSpecName "kube-api-access-j8h5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:57:11.094471 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.094067    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bca69aa4-8bb0-43b1-9113-82ad49cd5f6e" (UID: "bca69aa4-8bb0-43b1-9113-82ad49cd5f6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:57:11.094471 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.094286    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508b6f76-c334-4096-9d26-1c4000534efe-kube-api-access-l5qmj" (OuterVolumeSpecName: "kube-api-access-l5qmj") pod "508b6f76-c334-4096-9d26-1c4000534efe" (UID: "508b6f76-c334-4096-9d26-1c4000534efe"). InnerVolumeSpecName "kube-api-access-l5qmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:57:11.095513 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.095492    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-dshm" (OuterVolumeSpecName: "dshm") pod "508b6f76-c334-4096-9d26-1c4000534efe" (UID: "508b6f76-c334-4096-9d26-1c4000534efe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:11.190885 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.190856    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.190885 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.190882    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.191079 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.190894    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8h5v\" (UniqueName: \"kubernetes.io/projected/f8ea762c-f334-4f8e-baa2-03a183358369-kube-api-access-j8h5v\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.191079 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.190903    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftwp4\" (UniqueName: \"kubernetes.io/projected/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-kube-api-access-ftwp4\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.191079 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.190912    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.191079 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.190919    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.191079 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.190926    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508b6f76-c334-4096-9d26-1c4000534efe-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.191079 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.190934    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.191079 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.190940    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.191079 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.190947    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ea762c-f334-4f8e-baa2-03a183358369-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.191079 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.190954    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.191079 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.190962    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508b6f76-c334-4096-9d26-1c4000534efe-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.191079 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.190969    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17a22aae-0368-455b-9d83-308bc9760f26-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.191079 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.190975    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17a22aae-0368-455b-9d83-308bc9760f26-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.191079 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.190982    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ea762c-f334-4f8e-baa2-03a183358369-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.191079 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.190989    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5qmj\" (UniqueName: \"kubernetes.io/projected/508b6f76-c334-4096-9d26-1c4000534efe-kube-api-access-l5qmj\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.191079 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.190996    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.191079 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.191004    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68hf2\" (UniqueName: \"kubernetes.io/projected/17a22aae-0368-455b-9d83-308bc9760f26-kube-api-access-68hf2\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:11.340973 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.340942    2544 generic.go:356] "Generic (PLEG): container finished" podID="f8ea762c-f334-4f8e-baa2-03a183358369" containerID="6ebd0e3ec611f81bcceac85ab8bce12b198f0708813eff886e9d93956428603e" exitCode=0
Sep 11 21:57:11.341129 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.340994    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"
Sep 11 21:57:11.341129 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.341016    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2" event={"ID":"f8ea762c-f334-4f8e-baa2-03a183358369","Type":"ContainerDied","Data":"6ebd0e3ec611f81bcceac85ab8bce12b198f0708813eff886e9d93956428603e"}
Sep 11 21:57:11.341129 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.341045    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2" event={"ID":"f8ea762c-f334-4f8e-baa2-03a183358369","Type":"ContainerDied","Data":"60c92bccdeacfe78952b24f45e490294ebdce586e91b8e3942232057f306827c"}
Sep 11 21:57:11.341129 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.341061    2544 scope.go:117] "RemoveContainer" containerID="6ebd0e3ec611f81bcceac85ab8bce12b198f0708813eff886e9d93956428603e"
Sep 11 21:57:11.342357 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.342336    2544 generic.go:356] "Generic (PLEG): container finished" podID="bca69aa4-8bb0-43b1-9113-82ad49cd5f6e" containerID="d01f2d4ad6e599b2d69d67b0b7753165f5f1c6263581c1e555bb60376fed5485" exitCode=0
Sep 11 21:57:11.342455 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.342378    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"
Sep 11 21:57:11.342455 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.342410    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7" event={"ID":"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e","Type":"ContainerDied","Data":"d01f2d4ad6e599b2d69d67b0b7753165f5f1c6263581c1e555bb60376fed5485"}
Sep 11 21:57:11.342550 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.342459    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7" event={"ID":"bca69aa4-8bb0-43b1-9113-82ad49cd5f6e","Type":"ContainerDied","Data":"04a7b61b796113aaa11a7bd6c75e22de2086ac1ca2a5240055c7895cca2050e6"}
Sep 11 21:57:11.343804 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.343766    2544 generic.go:356] "Generic (PLEG): container finished" podID="508b6f76-c334-4096-9d26-1c4000534efe" containerID="5e530a8b0e9b7ff2c586ff6b36960de98928e8d8e75d89ee22252943fee82d3e" exitCode=0
Sep 11 21:57:11.343804 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.343801    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"
Sep 11 21:57:11.343979 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.343812    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs" event={"ID":"508b6f76-c334-4096-9d26-1c4000534efe","Type":"ContainerDied","Data":"5e530a8b0e9b7ff2c586ff6b36960de98928e8d8e75d89ee22252943fee82d3e"}
Sep 11 21:57:11.343979 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.343843    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs" event={"ID":"508b6f76-c334-4096-9d26-1c4000534efe","Type":"ContainerDied","Data":"11a93a8e7f9b1264be7e778577b08263497dfd7e78dc7b2adef3f15a697363e0"}
Sep 11 21:57:11.345359 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.345330    2544 generic.go:356] "Generic (PLEG): container finished" podID="17a22aae-0368-455b-9d83-308bc9760f26" containerID="da62f719acf07490586bbd29ecb1e1b87190ad21f739445c7e8d709227633647" exitCode=0
Sep 11 21:57:11.345472 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.345370    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r" event={"ID":"17a22aae-0368-455b-9d83-308bc9760f26","Type":"ContainerDied","Data":"da62f719acf07490586bbd29ecb1e1b87190ad21f739445c7e8d709227633647"}
Sep 11 21:57:11.345472 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.345395    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r" event={"ID":"17a22aae-0368-455b-9d83-308bc9760f26","Type":"ContainerDied","Data":"5ce0a4f01166cf4ccc145772414640c44d91fb7d3816878939d673df9108292b"}
Sep 11 21:57:11.345472 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.345399    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"
Sep 11 21:57:11.352001 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.351978    2544 scope.go:117] "RemoveContainer" containerID="6ebd0e3ec611f81bcceac85ab8bce12b198f0708813eff886e9d93956428603e"
Sep 11 21:57:11.352389 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:57:11.352343    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ebd0e3ec611f81bcceac85ab8bce12b198f0708813eff886e9d93956428603e\": container with ID starting with 6ebd0e3ec611f81bcceac85ab8bce12b198f0708813eff886e9d93956428603e not found: ID does not exist" containerID="6ebd0e3ec611f81bcceac85ab8bce12b198f0708813eff886e9d93956428603e"
Sep 11 21:57:11.352620 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.352377    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ebd0e3ec611f81bcceac85ab8bce12b198f0708813eff886e9d93956428603e"} err="failed to get container status \"6ebd0e3ec611f81bcceac85ab8bce12b198f0708813eff886e9d93956428603e\": rpc error: code = NotFound desc = could not find container \"6ebd0e3ec611f81bcceac85ab8bce12b198f0708813eff886e9d93956428603e\": container with ID starting with 6ebd0e3ec611f81bcceac85ab8bce12b198f0708813eff886e9d93956428603e not found: ID does not exist"
Sep 11 21:57:11.352620 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.352420    2544 scope.go:117] "RemoveContainer" containerID="d01f2d4ad6e599b2d69d67b0b7753165f5f1c6263581c1e555bb60376fed5485"
Sep 11 21:57:11.361957 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.361942    2544 scope.go:117] "RemoveContainer" containerID="d01f2d4ad6e599b2d69d67b0b7753165f5f1c6263581c1e555bb60376fed5485"
Sep 11 21:57:11.362197 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:57:11.362181    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d01f2d4ad6e599b2d69d67b0b7753165f5f1c6263581c1e555bb60376fed5485\": container with ID starting with d01f2d4ad6e599b2d69d67b0b7753165f5f1c6263581c1e555bb60376fed5485 not found: ID does not exist" containerID="d01f2d4ad6e599b2d69d67b0b7753165f5f1c6263581c1e555bb60376fed5485"
Sep 11 21:57:11.362270 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.362201    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01f2d4ad6e599b2d69d67b0b7753165f5f1c6263581c1e555bb60376fed5485"} err="failed to get container status \"d01f2d4ad6e599b2d69d67b0b7753165f5f1c6263581c1e555bb60376fed5485\": rpc error: code = NotFound desc = could not find container \"d01f2d4ad6e599b2d69d67b0b7753165f5f1c6263581c1e555bb60376fed5485\": container with ID starting with d01f2d4ad6e599b2d69d67b0b7753165f5f1c6263581c1e555bb60376fed5485 not found: ID does not exist"
Sep 11 21:57:11.362270 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.362227    2544 scope.go:117] "RemoveContainer" containerID="5e530a8b0e9b7ff2c586ff6b36960de98928e8d8e75d89ee22252943fee82d3e"
Sep 11 21:57:11.369360 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.369336    2544 scope.go:117] "RemoveContainer" containerID="5e530a8b0e9b7ff2c586ff6b36960de98928e8d8e75d89ee22252943fee82d3e"
Sep 11 21:57:11.369529 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.369511    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"]
Sep 11 21:57:11.369632 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:57:11.369615    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e530a8b0e9b7ff2c586ff6b36960de98928e8d8e75d89ee22252943fee82d3e\": container with ID starting with 5e530a8b0e9b7ff2c586ff6b36960de98928e8d8e75d89ee22252943fee82d3e not found: ID does not exist" containerID="5e530a8b0e9b7ff2c586ff6b36960de98928e8d8e75d89ee22252943fee82d3e"
Sep 11 21:57:11.369674 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.369639    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e530a8b0e9b7ff2c586ff6b36960de98928e8d8e75d89ee22252943fee82d3e"} err="failed to get container status \"5e530a8b0e9b7ff2c586ff6b36960de98928e8d8e75d89ee22252943fee82d3e\": rpc error: code = NotFound desc = could not find container \"5e530a8b0e9b7ff2c586ff6b36960de98928e8d8e75d89ee22252943fee82d3e\": container with ID starting with 5e530a8b0e9b7ff2c586ff6b36960de98928e8d8e75d89ee22252943fee82d3e not found: ID does not exist"
Sep 11 21:57:11.369674 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.369655    2544 scope.go:117] "RemoveContainer" containerID="da62f719acf07490586bbd29ecb1e1b87190ad21f739445c7e8d709227633647"
Sep 11 21:57:11.373493 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.373474    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/simulator-w4-35e777-kserve-56b754cc95-r75vs"]
Sep 11 21:57:11.376978 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.376963    2544 scope.go:117] "RemoveContainer" containerID="da62f719acf07490586bbd29ecb1e1b87190ad21f739445c7e8d709227633647"
Sep 11 21:57:11.377205 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:57:11.377189    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da62f719acf07490586bbd29ecb1e1b87190ad21f739445c7e8d709227633647\": container with ID starting with da62f719acf07490586bbd29ecb1e1b87190ad21f739445c7e8d709227633647 not found: ID does not exist" containerID="da62f719acf07490586bbd29ecb1e1b87190ad21f739445c7e8d709227633647"
Sep 11 21:57:11.377312 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.377211    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da62f719acf07490586bbd29ecb1e1b87190ad21f739445c7e8d709227633647"} err="failed to get container status \"da62f719acf07490586bbd29ecb1e1b87190ad21f739445c7e8d709227633647\": rpc error: code = NotFound desc = could not find container \"da62f719acf07490586bbd29ecb1e1b87190ad21f739445c7e8d709227633647\": container with ID starting with da62f719acf07490586bbd29ecb1e1b87190ad21f739445c7e8d709227633647 not found: ID does not exist"
Sep 11 21:57:11.383846 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.383826    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"]
Sep 11 21:57:11.387735 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.387716    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/distinct-2-w4-35e777-kserve-5dcd4fd4cd-8vs8r"]
Sep 11 21:57:11.397607 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.397589    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"]
Sep 11 21:57:11.401814 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.401795    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/premium-w4-35e777-kserve-b8cfc7758-qtcn2"]
Sep 11 21:57:11.410721 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.410702    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"]
Sep 11 21:57:11.414966 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.414946    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-35e777/distinct-w4-35e777-kserve-6fd77c5b77-srbn7"]
Sep 11 21:57:11.593732 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.593655    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17a22aae-0368-455b-9d83-308bc9760f26" path="/var/lib/kubelet/pods/17a22aae-0368-455b-9d83-308bc9760f26/volumes"
Sep 11 21:57:11.594178 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.594160    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="508b6f76-c334-4096-9d26-1c4000534efe" path="/var/lib/kubelet/pods/508b6f76-c334-4096-9d26-1c4000534efe/volumes"
Sep 11 21:57:11.594522 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.594507    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b44d7aba-a895-4292-bbb9-ee87bc6952fb" path="/var/lib/kubelet/pods/b44d7aba-a895-4292-bbb9-ee87bc6952fb/volumes"
Sep 11 21:57:11.594914 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.594895    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bca69aa4-8bb0-43b1-9113-82ad49cd5f6e" path="/var/lib/kubelet/pods/bca69aa4-8bb0-43b1-9113-82ad49cd5f6e/volumes"
Sep 11 21:57:11.595399 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:11.595385    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8ea762c-f334-4f8e-baa2-03a183358369" path="/var/lib/kubelet/pods/f8ea762c-f334-4f8e-baa2-03a183358369/volumes"
Sep 11 21:57:17.022563 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:17.022526    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"]
Sep 11 21:57:17.023064 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:17.022771    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq" podUID="1a20e93f-ea55-42d2-9f62-1b971c171d7b" containerName="istio-proxy" containerID="cri-o://d681eac6b494106c20830ed9966ad29a39465546730bcc9de48e7d2be3f77dc6" gracePeriod=30
Sep 11 21:57:18.971250 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:18.971203    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"]
Sep 11 21:57:18.971654 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:18.971410    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8" podUID="fb93698f-4a48-4b60-be78-74623c953109" containerName="istio-proxy" containerID="cri-o://1f0601e877b3b9a18c043558fdebd620504500b7c4f4e7f23bee52e941007fcd" gracePeriod=30
Sep 11 21:57:22.268989 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.268964    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:57:22.394860 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.394780    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-envoy\") pod \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") "
Sep 11 21:57:22.394860 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.394814    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-data\") pod \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") "
Sep 11 21:57:22.394860 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.394837    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istiod-ca-cert\") pod \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") "
Sep 11 21:57:22.395143 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.394879    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-token\") pod \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") "
Sep 11 21:57:22.395143 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.394916    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf2z9\" (UniqueName: \"kubernetes.io/projected/1a20e93f-ea55-42d2-9f62-1b971c171d7b-kube-api-access-gf2z9\") pod \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") "
Sep 11 21:57:22.395143 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.394939    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-podinfo\") pod \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") "
Sep 11 21:57:22.395143 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.394960    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-workload-certs\") pod \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") "
Sep 11 21:57:22.395143 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.394983    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-workload-socket\") pod \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") "
Sep 11 21:57:22.395143 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.395066    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-credential-socket\") pod \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\" (UID: \"1a20e93f-ea55-42d2-9f62-1b971c171d7b\") "
Sep 11 21:57:22.395143 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.395101    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a20e93f-ea55-42d2-9f62-1b971c171d7b" (UID: "1a20e93f-ea55-42d2-9f62-1b971c171d7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:22.395522 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.395314    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a20e93f-ea55-42d2-9f62-1b971c171d7b" (UID: "1a20e93f-ea55-42d2-9f62-1b971c171d7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:22.395522 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.395351    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a20e93f-ea55-42d2-9f62-1b971c171d7b" (UID: "1a20e93f-ea55-42d2-9f62-1b971c171d7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:22.395522 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.395383    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a20e93f-ea55-42d2-9f62-1b971c171d7b" (UID: "1a20e93f-ea55-42d2-9f62-1b971c171d7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:57:22.395522 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.395461    2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-data\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:22.395522 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.395478    2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-workload-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:22.395522 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.395522    2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-workload-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:22.395839 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.395569    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a20e93f-ea55-42d2-9f62-1b971c171d7b" (UID: "1a20e93f-ea55-42d2-9f62-1b971c171d7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:22.397158 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.397133    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a20e93f-ea55-42d2-9f62-1b971c171d7b" (UID: "1a20e93f-ea55-42d2-9f62-1b971c171d7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:22.397277 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.397137    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a20e93f-ea55-42d2-9f62-1b971c171d7b" (UID: "1a20e93f-ea55-42d2-9f62-1b971c171d7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:57:22.397277 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.397161    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a20e93f-ea55-42d2-9f62-1b971c171d7b-kube-api-access-gf2z9" (OuterVolumeSpecName: "kube-api-access-gf2z9") pod "1a20e93f-ea55-42d2-9f62-1b971c171d7b" (UID: "1a20e93f-ea55-42d2-9f62-1b971c171d7b"). InnerVolumeSpecName "kube-api-access-gf2z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:57:22.397277 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.397168    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a20e93f-ea55-42d2-9f62-1b971c171d7b" (UID: "1a20e93f-ea55-42d2-9f62-1b971c171d7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:57:22.407184 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.407157    2544 generic.go:356] "Generic (PLEG): container finished" podID="1a20e93f-ea55-42d2-9f62-1b971c171d7b" containerID="d681eac6b494106c20830ed9966ad29a39465546730bcc9de48e7d2be3f77dc6" exitCode=0
Sep 11 21:57:22.407274 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.407196    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"
Sep 11 21:57:22.407274 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.407212    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq" event={"ID":"1a20e93f-ea55-42d2-9f62-1b971c171d7b","Type":"ContainerDied","Data":"d681eac6b494106c20830ed9966ad29a39465546730bcc9de48e7d2be3f77dc6"}
Sep 11 21:57:22.407364 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.407292    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq" event={"ID":"1a20e93f-ea55-42d2-9f62-1b971c171d7b","Type":"ContainerDied","Data":"8f275d83e69f8575e178522e181d9b93b4e83a630e8f7e108cd1a724e6527c58"}
Sep 11 21:57:22.407364 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.407326    2544 scope.go:117] "RemoveContainer" containerID="d681eac6b494106c20830ed9966ad29a39465546730bcc9de48e7d2be3f77dc6"
Sep 11 21:57:22.417697 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.417678    2544 scope.go:117] "RemoveContainer" containerID="954d78cdd9743fddfdc7db61c96d687fb455bd73066f0fb7016f940e8c3db64c"
Sep 11 21:57:22.425203 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.425189    2544 scope.go:117] "RemoveContainer" containerID="d681eac6b494106c20830ed9966ad29a39465546730bcc9de48e7d2be3f77dc6"
Sep 11 21:57:22.425488 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:57:22.425469    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d681eac6b494106c20830ed9966ad29a39465546730bcc9de48e7d2be3f77dc6\": container with ID starting with d681eac6b494106c20830ed9966ad29a39465546730bcc9de48e7d2be3f77dc6 not found: ID does not exist" containerID="d681eac6b494106c20830ed9966ad29a39465546730bcc9de48e7d2be3f77dc6"
Sep 11 21:57:22.425531 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.425497    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d681eac6b494106c20830ed9966ad29a39465546730bcc9de48e7d2be3f77dc6"} err="failed to get container status \"d681eac6b494106c20830ed9966ad29a39465546730bcc9de48e7d2be3f77dc6\": rpc error: code = NotFound desc = could not find container \"d681eac6b494106c20830ed9966ad29a39465546730bcc9de48e7d2be3f77dc6\": container with ID starting with d681eac6b494106c20830ed9966ad29a39465546730bcc9de48e7d2be3f77dc6 not found: ID does not exist"
Sep 11 21:57:22.425531 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.425512    2544 scope.go:117] "RemoveContainer" containerID="954d78cdd9743fddfdc7db61c96d687fb455bd73066f0fb7016f940e8c3db64c"
Sep 11 21:57:22.425758 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:57:22.425742    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"954d78cdd9743fddfdc7db61c96d687fb455bd73066f0fb7016f940e8c3db64c\": container with ID starting with 954d78cdd9743fddfdc7db61c96d687fb455bd73066f0fb7016f940e8c3db64c not found: ID does not exist" containerID="954d78cdd9743fddfdc7db61c96d687fb455bd73066f0fb7016f940e8c3db64c"
Sep 11 21:57:22.425802 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.425762    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"954d78cdd9743fddfdc7db61c96d687fb455bd73066f0fb7016f940e8c3db64c"} err="failed to get container status \"954d78cdd9743fddfdc7db61c96d687fb455bd73066f0fb7016f940e8c3db64c\": rpc error: code = NotFound desc = could not find container \"954d78cdd9743fddfdc7db61c96d687fb455bd73066f0fb7016f940e8c3db64c\": container with ID starting with 954d78cdd9743fddfdc7db61c96d687fb455bd73066f0fb7016f940e8c3db64c not found: ID does not exist"
Sep 11 21:57:22.428814 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.428795    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"]
Sep 11 21:57:22.433018 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.433001    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-35e777-openshift-default-775c996888-28smq"]
Sep 11 21:57:22.496047 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.496022    2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-credential-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:22.496047 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.496048    2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-envoy\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:22.496195 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.496062    2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istiod-ca-cert\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:22.496195 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.496074    2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-token\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:22.496195 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.496087    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf2z9\" (UniqueName: \"kubernetes.io/projected/1a20e93f-ea55-42d2-9f62-1b971c171d7b-kube-api-access-gf2z9\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:22.496195 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:22.496099    2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a20e93f-ea55-42d2-9f62-1b971c171d7b-istio-podinfo\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:23.591836 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:23.591804    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a20e93f-ea55-42d2-9f62-1b971c171d7b" path="/var/lib/kubelet/pods/1a20e93f-ea55-42d2-9f62-1b971c171d7b/volumes"
Sep 11 21:57:24.225084 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.225054    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:57:24.313496 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.313402    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-istio-data\") pod \"fb93698f-4a48-4b60-be78-74623c953109\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") "
Sep 11 21:57:24.313496 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.313461    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj25k\" (UniqueName: \"kubernetes.io/projected/fb93698f-4a48-4b60-be78-74623c953109-kube-api-access-fj25k\") pod \"fb93698f-4a48-4b60-be78-74623c953109\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") "
Sep 11 21:57:24.313738 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.313509    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb93698f-4a48-4b60-be78-74623c953109-istio-podinfo\") pod \"fb93698f-4a48-4b60-be78-74623c953109\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") "
Sep 11 21:57:24.313738 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.313540    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb93698f-4a48-4b60-be78-74623c953109-istiod-ca-cert\") pod \"fb93698f-4a48-4b60-be78-74623c953109\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") "
Sep 11 21:57:24.313738 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.313562    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb93698f-4a48-4b60-be78-74623c953109-istio-token\") pod \"fb93698f-4a48-4b60-be78-74623c953109\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") "
Sep 11 21:57:24.313738 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.313684    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-workload-socket\") pod \"fb93698f-4a48-4b60-be78-74623c953109\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") "
Sep 11 21:57:24.313943 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.313751    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-istio-envoy\") pod \"fb93698f-4a48-4b60-be78-74623c953109\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") "
Sep 11 21:57:24.313943 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.313754    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-istio-data" (OuterVolumeSpecName: "istio-data") pod "fb93698f-4a48-4b60-be78-74623c953109" (UID: "fb93698f-4a48-4b60-be78-74623c953109"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:24.313943 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.313827    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-credential-socket\") pod \"fb93698f-4a48-4b60-be78-74623c953109\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") "
Sep 11 21:57:24.313943 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.313892    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-workload-certs\") pod \"fb93698f-4a48-4b60-be78-74623c953109\" (UID: \"fb93698f-4a48-4b60-be78-74623c953109\") "
Sep 11 21:57:24.314146 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.313962    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fb93698f-4a48-4b60-be78-74623c953109" (UID: "fb93698f-4a48-4b60-be78-74623c953109"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:24.314146 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.314041    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb93698f-4a48-4b60-be78-74623c953109-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fb93698f-4a48-4b60-be78-74623c953109" (UID: "fb93698f-4a48-4b60-be78-74623c953109"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:57:24.314146 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.314069    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fb93698f-4a48-4b60-be78-74623c953109" (UID: "fb93698f-4a48-4b60-be78-74623c953109"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:24.314350 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.314239    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fb93698f-4a48-4b60-be78-74623c953109" (UID: "fb93698f-4a48-4b60-be78-74623c953109"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:24.314350 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.314243    2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb93698f-4a48-4b60-be78-74623c953109-istiod-ca-cert\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:24.314350 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.314286    2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-workload-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:24.314350 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.314304    2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-credential-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:24.314350 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.314318    2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-istio-data\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:24.315803 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.315781    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb93698f-4a48-4b60-be78-74623c953109-istio-token" (OuterVolumeSpecName: "istio-token") pod "fb93698f-4a48-4b60-be78-74623c953109" (UID: "fb93698f-4a48-4b60-be78-74623c953109"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:57:24.315911 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.315799    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb93698f-4a48-4b60-be78-74623c953109-kube-api-access-fj25k" (OuterVolumeSpecName: "kube-api-access-fj25k") pod "fb93698f-4a48-4b60-be78-74623c953109" (UID: "fb93698f-4a48-4b60-be78-74623c953109"). InnerVolumeSpecName "kube-api-access-fj25k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:57:24.315911 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.315814    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fb93698f-4a48-4b60-be78-74623c953109" (UID: "fb93698f-4a48-4b60-be78-74623c953109"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:24.315911 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.315802    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb93698f-4a48-4b60-be78-74623c953109-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fb93698f-4a48-4b60-be78-74623c953109" (UID: "fb93698f-4a48-4b60-be78-74623c953109"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:57:24.414682 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.414661    2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb93698f-4a48-4b60-be78-74623c953109-istio-token\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:24.414682 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.414681    2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-istio-envoy\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:24.414840 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.414690    2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb93698f-4a48-4b60-be78-74623c953109-workload-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:24.414840 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.414699    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj25k\" (UniqueName: \"kubernetes.io/projected/fb93698f-4a48-4b60-be78-74623c953109-kube-api-access-fj25k\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:24.414840 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.414708    2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb93698f-4a48-4b60-be78-74623c953109-istio-podinfo\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:57:24.420448 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.420422    2544 generic.go:356] "Generic (PLEG): container finished" podID="fb93698f-4a48-4b60-be78-74623c953109" containerID="1f0601e877b3b9a18c043558fdebd620504500b7c4f4e7f23bee52e941007fcd" exitCode=0
Sep 11 21:57:24.420575 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.420461    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8" event={"ID":"fb93698f-4a48-4b60-be78-74623c953109","Type":"ContainerDied","Data":"1f0601e877b3b9a18c043558fdebd620504500b7c4f4e7f23bee52e941007fcd"}
Sep 11 21:57:24.420575 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.420491    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"
Sep 11 21:57:24.420575 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.420544    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8" event={"ID":"fb93698f-4a48-4b60-be78-74623c953109","Type":"ContainerDied","Data":"370a96d82ac9f40cbda949581ceafc113818fea37180536678e5432796f339f4"}
Sep 11 21:57:24.420575 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.420568    2544 scope.go:117] "RemoveContainer" containerID="1f0601e877b3b9a18c043558fdebd620504500b7c4f4e7f23bee52e941007fcd"
Sep 11 21:57:24.429478 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.429456    2544 scope.go:117] "RemoveContainer" containerID="34d57f3755beb1181a142760f748eeebd59288b35f07afef2bf4957d195bc008"
Sep 11 21:57:24.436984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.436964    2544 scope.go:117] "RemoveContainer" containerID="1f0601e877b3b9a18c043558fdebd620504500b7c4f4e7f23bee52e941007fcd"
Sep 11 21:57:24.437274 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:57:24.437249    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f0601e877b3b9a18c043558fdebd620504500b7c4f4e7f23bee52e941007fcd\": container with ID starting with 1f0601e877b3b9a18c043558fdebd620504500b7c4f4e7f23bee52e941007fcd not found: ID does not exist" containerID="1f0601e877b3b9a18c043558fdebd620504500b7c4f4e7f23bee52e941007fcd"
Sep 11 21:57:24.437343 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.437285    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f0601e877b3b9a18c043558fdebd620504500b7c4f4e7f23bee52e941007fcd"} err="failed to get container status \"1f0601e877b3b9a18c043558fdebd620504500b7c4f4e7f23bee52e941007fcd\": rpc error: code = NotFound desc = could not find container \"1f0601e877b3b9a18c043558fdebd620504500b7c4f4e7f23bee52e941007fcd\": container with ID starting with 1f0601e877b3b9a18c043558fdebd620504500b7c4f4e7f23bee52e941007fcd not found: ID does not exist"
Sep 11 21:57:24.437343 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.437308    2544 scope.go:117] "RemoveContainer" containerID="34d57f3755beb1181a142760f748eeebd59288b35f07afef2bf4957d195bc008"
Sep 11 21:57:24.437606 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:57:24.437582    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d57f3755beb1181a142760f748eeebd59288b35f07afef2bf4957d195bc008\": container with ID starting with 34d57f3755beb1181a142760f748eeebd59288b35f07afef2bf4957d195bc008 not found: ID does not exist" containerID="34d57f3755beb1181a142760f748eeebd59288b35f07afef2bf4957d195bc008"
Sep 11 21:57:24.437657 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.437615    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d57f3755beb1181a142760f748eeebd59288b35f07afef2bf4957d195bc008"} err="failed to get container status \"34d57f3755beb1181a142760f748eeebd59288b35f07afef2bf4957d195bc008\": rpc error: code = NotFound desc = could not find container \"34d57f3755beb1181a142760f748eeebd59288b35f07afef2bf4957d195bc008\": container with ID starting with 34d57f3755beb1181a142760f748eeebd59288b35f07afef2bf4957d195bc008 not found: ID does not exist"
Sep 11 21:57:24.441612 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.441590    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"]
Sep 11 21:57:24.444779 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:24.444761    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-afd794-openshift-default-688d446bcd-6bcv8"]
Sep 11 21:57:25.591102 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:25.591071    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb93698f-4a48-4b60-be78-74623c953109" path="/var/lib/kubelet/pods/fb93698f-4a48-4b60-be78-74623c953109/volumes"
Sep 11 21:57:37.835267 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:37.835181    2544 scope.go:117] "RemoveContainer" containerID="6802469d941e81b22628fe519d27ad928325429b68da6c585044de4b5abab8e3"
Sep 11 21:57:37.843497 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:57:37.843476    2544 scope.go:117] "RemoveContainer" containerID="d54f0b6ddab101409896991e48993633f85eeaa55eb2de66707c18e9bd9cc28d"
Sep 11 21:58:06.295275 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:06.295241    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"]
Sep 11 21:58:12.068475 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.068437    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"]
Sep 11 21:58:12.068938 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.068919    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a20e93f-ea55-42d2-9f62-1b971c171d7b" containerName="istio-proxy"
Sep 11 21:58:12.068984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.068942    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a20e93f-ea55-42d2-9f62-1b971c171d7b" containerName="istio-proxy"
Sep 11 21:58:12.068984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.068954    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca69aa4-8bb0-43b1-9113-82ad49cd5f6e" containerName="main"
Sep 11 21:58:12.068984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.068963    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca69aa4-8bb0-43b1-9113-82ad49cd5f6e" containerName="main"
Sep 11 21:58:12.068984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.068973    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="508b6f76-c334-4096-9d26-1c4000534efe" containerName="main"
Sep 11 21:58:12.068984 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.068981    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="508b6f76-c334-4096-9d26-1c4000534efe" containerName="main"
Sep 11 21:58:12.069137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.068997    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb93698f-4a48-4b60-be78-74623c953109" containerName="istio-proxy"
Sep 11 21:58:12.069137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069006    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb93698f-4a48-4b60-be78-74623c953109" containerName="istio-proxy"
Sep 11 21:58:12.069137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069021    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a20e93f-ea55-42d2-9f62-1b971c171d7b" containerName="istio-proxy"
Sep 11 21:58:12.069137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069029    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a20e93f-ea55-42d2-9f62-1b971c171d7b" containerName="istio-proxy"
Sep 11 21:58:12.069137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069040    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44d7aba-a895-4292-bbb9-ee87bc6952fb" containerName="revoke-keys"
Sep 11 21:58:12.069137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069049    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44d7aba-a895-4292-bbb9-ee87bc6952fb" containerName="revoke-keys"
Sep 11 21:58:12.069137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069064    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8ea762c-f334-4f8e-baa2-03a183358369" containerName="main"
Sep 11 21:58:12.069137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069072    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8ea762c-f334-4f8e-baa2-03a183358369" containerName="main"
Sep 11 21:58:12.069137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069080    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb93698f-4a48-4b60-be78-74623c953109" containerName="istio-proxy"
Sep 11 21:58:12.069137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069088    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb93698f-4a48-4b60-be78-74623c953109" containerName="istio-proxy"
Sep 11 21:58:12.069137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069100    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a22aae-0368-455b-9d83-308bc9760f26" containerName="main"
Sep 11 21:58:12.069137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069108    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a22aae-0368-455b-9d83-308bc9760f26" containerName="main"
Sep 11 21:58:12.069137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069123    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e595e7-a303-4af1-b7ba-cd43a4407b99" containerName="istio-proxy"
Sep 11 21:58:12.069137 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069131    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e595e7-a303-4af1-b7ba-cd43a4407b99" containerName="istio-proxy"
Sep 11 21:58:12.069606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069239    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb93698f-4a48-4b60-be78-74623c953109" containerName="istio-proxy"
Sep 11 21:58:12.069606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069252    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44d7aba-a895-4292-bbb9-ee87bc6952fb" containerName="revoke-keys"
Sep 11 21:58:12.069606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069263    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a22aae-0368-455b-9d83-308bc9760f26" containerName="main"
Sep 11 21:58:12.069606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069273    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="40e595e7-a303-4af1-b7ba-cd43a4407b99" containerName="istio-proxy"
Sep 11 21:58:12.069606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069284    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="508b6f76-c334-4096-9d26-1c4000534efe" containerName="main"
Sep 11 21:58:12.069606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069293    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a20e93f-ea55-42d2-9f62-1b971c171d7b" containerName="istio-proxy"
Sep 11 21:58:12.069606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069301    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a20e93f-ea55-42d2-9f62-1b971c171d7b" containerName="istio-proxy"
Sep 11 21:58:12.069606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069311    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8ea762c-f334-4f8e-baa2-03a183358369" containerName="main"
Sep 11 21:58:12.069606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069318    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="bca69aa4-8bb0-43b1-9113-82ad49cd5f6e" containerName="main"
Sep 11 21:58:12.069606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.069326    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb93698f-4a48-4b60-be78-74623c953109" containerName="istio-proxy"
Sep 11 21:58:12.072503 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.072487    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.074980 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.074956    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7d7821\"/\"sub-test-model-7d7821-kserve-self-signed-certs\""
Sep 11 21:58:12.078255 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.078210    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"]
Sep 11 21:58:12.132434 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.132405    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-home\") pod \"sub-test-model-7d7821-kserve-68645548f7-5ds97\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.132566 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.132439    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-dshm\") pod \"sub-test-model-7d7821-kserve-68645548f7-5ds97\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.132566 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.132459    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-model-cache\") pod \"sub-test-model-7d7821-kserve-68645548f7-5ds97\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.132566 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.132485    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjkhk\" (UniqueName: \"kubernetes.io/projected/4d4edd91-13c2-433c-9211-cbeb721d2b7c-kube-api-access-cjkhk\") pod \"sub-test-model-7d7821-kserve-68645548f7-5ds97\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.132566 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.132518    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4edd91-13c2-433c-9211-cbeb721d2b7c-tls-certs\") pod \"sub-test-model-7d7821-kserve-68645548f7-5ds97\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.132739 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.132602    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-tmp-dir\") pod \"sub-test-model-7d7821-kserve-68645548f7-5ds97\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.233485 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.233448    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjkhk\" (UniqueName: \"kubernetes.io/projected/4d4edd91-13c2-433c-9211-cbeb721d2b7c-kube-api-access-cjkhk\") pod \"sub-test-model-7d7821-kserve-68645548f7-5ds97\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.233485 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.233488    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4edd91-13c2-433c-9211-cbeb721d2b7c-tls-certs\") pod \"sub-test-model-7d7821-kserve-68645548f7-5ds97\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.233748 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.233545    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-tmp-dir\") pod \"sub-test-model-7d7821-kserve-68645548f7-5ds97\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.233748 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.233591    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-home\") pod \"sub-test-model-7d7821-kserve-68645548f7-5ds97\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.233748 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.233613    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-dshm\") pod \"sub-test-model-7d7821-kserve-68645548f7-5ds97\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.233748 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.233634    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-model-cache\") pod \"sub-test-model-7d7821-kserve-68645548f7-5ds97\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.233961 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.233943    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-tmp-dir\") pod \"sub-test-model-7d7821-kserve-68645548f7-5ds97\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.234019 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.233993    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-home\") pod \"sub-test-model-7d7821-kserve-68645548f7-5ds97\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.234068 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.234029    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-model-cache\") pod \"sub-test-model-7d7821-kserve-68645548f7-5ds97\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.239237 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.236249    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-dshm\") pod \"sub-test-model-7d7821-kserve-68645548f7-5ds97\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.239237 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.236355    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4edd91-13c2-433c-9211-cbeb721d2b7c-tls-certs\") pod \"sub-test-model-7d7821-kserve-68645548f7-5ds97\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.244951 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.244925    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjkhk\" (UniqueName: \"kubernetes.io/projected/4d4edd91-13c2-433c-9211-cbeb721d2b7c-kube-api-access-cjkhk\") pod \"sub-test-model-7d7821-kserve-68645548f7-5ds97\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.382872 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.382784    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:12.651442 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:12.651414    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"]
Sep 11 21:58:12.653687 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:58:12.653648    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d4edd91_13c2_433c_9211_cbeb721d2b7c.slice/crio-00e7b1908a9d12f8cf2376acb215ea7bff6164444e15fcc446a53b0955b007de WatchSource:0}: Error finding container 00e7b1908a9d12f8cf2376acb215ea7bff6164444e15fcc446a53b0955b007de: Status 404 returned error can't find the container with id 00e7b1908a9d12f8cf2376acb215ea7bff6164444e15fcc446a53b0955b007de
Sep 11 21:58:13.649259 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:13.649192    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97" event={"ID":"4d4edd91-13c2-433c-9211-cbeb721d2b7c","Type":"ContainerStarted","Data":"3d6b6224f30b99684e130ae8aa6a5b9f29e5d5665b2aa9bb89cd5ee33c875506"}
Sep 11 21:58:13.649678 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:13.649271    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97" event={"ID":"4d4edd91-13c2-433c-9211-cbeb721d2b7c","Type":"ContainerStarted","Data":"00e7b1908a9d12f8cf2376acb215ea7bff6164444e15fcc446a53b0955b007de"}
Sep 11 21:58:13.666303 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:13.666251    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97" podStartSLOduration=1.666233176 podStartE2EDuration="1.666233176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:58:13.665507938 +0000 UTC m=+1656.634924757" watchObservedRunningTime="2026-09-11 21:58:13.666233176 +0000 UTC m=+1656.635649990"
Sep 11 21:58:21.313903 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.313839    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b" podUID="3882d1cd-226a-4def-a263-888579879ff6" containerName="main" containerID="cri-o://2c2437a3261adb652eb740abf51680fa2f17c802e7797eb140366d27d84aea18" gracePeriod=45
Sep 11 21:58:21.564364 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.564304    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:58:21.690375 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.690333    2544 generic.go:356] "Generic (PLEG): container finished" podID="3882d1cd-226a-4def-a263-888579879ff6" containerID="2c2437a3261adb652eb740abf51680fa2f17c802e7797eb140366d27d84aea18" exitCode=0
Sep 11 21:58:21.690375 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.690376    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"
Sep 11 21:58:21.690375 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.690380    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b" event={"ID":"3882d1cd-226a-4def-a263-888579879ff6","Type":"ContainerDied","Data":"2c2437a3261adb652eb740abf51680fa2f17c802e7797eb140366d27d84aea18"}
Sep 11 21:58:21.690622 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.690419    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b" event={"ID":"3882d1cd-226a-4def-a263-888579879ff6","Type":"ContainerDied","Data":"c2d0dc5e2b8ef162526ceaed95c21d9c45b9604bf0b07a0dbd20f8a177810d7e"}
Sep 11 21:58:21.690622 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.690442    2544 scope.go:117] "RemoveContainer" containerID="2c2437a3261adb652eb740abf51680fa2f17c802e7797eb140366d27d84aea18"
Sep 11 21:58:21.698712 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.698680    2544 scope.go:117] "RemoveContainer" containerID="2c2437a3261adb652eb740abf51680fa2f17c802e7797eb140366d27d84aea18"
Sep 11 21:58:21.699002 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:58:21.698982    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c2437a3261adb652eb740abf51680fa2f17c802e7797eb140366d27d84aea18\": container with ID starting with 2c2437a3261adb652eb740abf51680fa2f17c802e7797eb140366d27d84aea18 not found: ID does not exist" containerID="2c2437a3261adb652eb740abf51680fa2f17c802e7797eb140366d27d84aea18"
Sep 11 21:58:21.699064 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.699012    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2437a3261adb652eb740abf51680fa2f17c802e7797eb140366d27d84aea18"} err="failed to get container status \"2c2437a3261adb652eb740abf51680fa2f17c802e7797eb140366d27d84aea18\": rpc error: code = NotFound desc = could not find container \"2c2437a3261adb652eb740abf51680fa2f17c802e7797eb140366d27d84aea18\": container with ID starting with 2c2437a3261adb652eb740abf51680fa2f17c802e7797eb140366d27d84aea18 not found: ID does not exist"
Sep 11 21:58:21.717307 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.717285    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-dshm\") pod \"3882d1cd-226a-4def-a263-888579879ff6\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") "
Sep 11 21:58:21.717407 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.717360    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3882d1cd-226a-4def-a263-888579879ff6-tls-certs\") pod \"3882d1cd-226a-4def-a263-888579879ff6\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") "
Sep 11 21:58:21.717407 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.717389    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9v84d\" (UniqueName: \"kubernetes.io/projected/3882d1cd-226a-4def-a263-888579879ff6-kube-api-access-9v84d\") pod \"3882d1cd-226a-4def-a263-888579879ff6\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") "
Sep 11 21:58:21.717546 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.717528    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-tmp-dir\") pod \"3882d1cd-226a-4def-a263-888579879ff6\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") "
Sep 11 21:58:21.717606 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.717575    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-model-cache\") pod \"3882d1cd-226a-4def-a263-888579879ff6\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") "
Sep 11 21:58:21.717664 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.717634    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-home\") pod \"3882d1cd-226a-4def-a263-888579879ff6\" (UID: \"3882d1cd-226a-4def-a263-888579879ff6\") "
Sep 11 21:58:21.717862 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.717797    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3882d1cd-226a-4def-a263-888579879ff6" (UID: "3882d1cd-226a-4def-a263-888579879ff6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:58:21.717976 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.717908    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-model-cache" (OuterVolumeSpecName: "model-cache") pod "3882d1cd-226a-4def-a263-888579879ff6" (UID: "3882d1cd-226a-4def-a263-888579879ff6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:58:21.717976 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.717937    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-home" (OuterVolumeSpecName: "home") pod "3882d1cd-226a-4def-a263-888579879ff6" (UID: "3882d1cd-226a-4def-a263-888579879ff6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:58:21.718094 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.718003    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:58:21.718094 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.718015    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:58:21.718094 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.718024    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:58:21.719505 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.719482    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-dshm" (OuterVolumeSpecName: "dshm") pod "3882d1cd-226a-4def-a263-888579879ff6" (UID: "3882d1cd-226a-4def-a263-888579879ff6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:58:21.719584 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.719568    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3882d1cd-226a-4def-a263-888579879ff6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3882d1cd-226a-4def-a263-888579879ff6" (UID: "3882d1cd-226a-4def-a263-888579879ff6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:58:21.719631 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.719583    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3882d1cd-226a-4def-a263-888579879ff6-kube-api-access-9v84d" (OuterVolumeSpecName: "kube-api-access-9v84d") pod "3882d1cd-226a-4def-a263-888579879ff6" (UID: "3882d1cd-226a-4def-a263-888579879ff6"). InnerVolumeSpecName "kube-api-access-9v84d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:58:21.818684 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.818593    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3882d1cd-226a-4def-a263-888579879ff6-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:58:21.818684 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.818623    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v84d\" (UniqueName: \"kubernetes.io/projected/3882d1cd-226a-4def-a263-888579879ff6-kube-api-access-9v84d\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:58:21.818684 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:21.818638    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3882d1cd-226a-4def-a263-888579879ff6-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:58:22.011586 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:22.011551    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"]
Sep 11 21:58:22.015285 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:22.015251    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7d7821/auth-test-model-7d7821-kserve-7f459694d7-b845b"]
Sep 11 21:58:22.383043 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:22.383012    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:22.383550 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:22.383062    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:22.395446 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:22.395423    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:22.710861 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:22.710774    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:58:23.594045 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:23.594010    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3882d1cd-226a-4def-a263-888579879ff6" path="/var/lib/kubelet/pods/3882d1cd-226a-4def-a263-888579879ff6/volumes"
Sep 11 21:58:30.647825 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.647787    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"]
Sep 11 21:58:30.648277 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.648109    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3882d1cd-226a-4def-a263-888579879ff6" containerName="main"
Sep 11 21:58:30.648277 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.648121    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="3882d1cd-226a-4def-a263-888579879ff6" containerName="main"
Sep 11 21:58:30.648277 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.648183    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="3882d1cd-226a-4def-a263-888579879ff6" containerName="main"
Sep 11 21:58:30.653767 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.653730    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.657676 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.657644    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4e0323\"/\"default-dockercfg-lp4bk\""
Sep 11 21:58:30.657876 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.657853    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4e0323\"/\"openshift-service-ca.crt\""
Sep 11 21:58:30.658906 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.658886    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4e0323\"/\"sub-test-model-4e0323-kserve-self-signed-certs\""
Sep 11 21:58:30.659054 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.658916    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4e0323\"/\"kube-root-ca.crt\""
Sep 11 21:58:30.661508 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.661486    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"]
Sep 11 21:58:30.702636 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.702600    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phckp\" (UniqueName: \"kubernetes.io/projected/74f27198-ed0b-4261-9194-4a6900a1ff01-kube-api-access-phckp\") pod \"sub-test-model-4e0323-kserve-56cf4c674f-c48fv\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.702801 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.702642    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-tmp-dir\") pod \"sub-test-model-4e0323-kserve-56cf4c674f-c48fv\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.702801 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.702668    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-home\") pod \"sub-test-model-4e0323-kserve-56cf4c674f-c48fv\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.702801 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.702694    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f27198-ed0b-4261-9194-4a6900a1ff01-tls-certs\") pod \"sub-test-model-4e0323-kserve-56cf4c674f-c48fv\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.702801 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.702789    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-model-cache\") pod \"sub-test-model-4e0323-kserve-56cf4c674f-c48fv\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.702941 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.702827    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-dshm\") pod \"sub-test-model-4e0323-kserve-56cf4c674f-c48fv\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.804104 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.804062    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phckp\" (UniqueName: \"kubernetes.io/projected/74f27198-ed0b-4261-9194-4a6900a1ff01-kube-api-access-phckp\") pod \"sub-test-model-4e0323-kserve-56cf4c674f-c48fv\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.804328 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.804108    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-tmp-dir\") pod \"sub-test-model-4e0323-kserve-56cf4c674f-c48fv\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.804328 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.804147    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-home\") pod \"sub-test-model-4e0323-kserve-56cf4c674f-c48fv\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.804328 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.804178    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f27198-ed0b-4261-9194-4a6900a1ff01-tls-certs\") pod \"sub-test-model-4e0323-kserve-56cf4c674f-c48fv\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.804328 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.804248    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-model-cache\") pod \"sub-test-model-4e0323-kserve-56cf4c674f-c48fv\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.804328 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.804293    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-dshm\") pod \"sub-test-model-4e0323-kserve-56cf4c674f-c48fv\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.804631 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.804605    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-tmp-dir\") pod \"sub-test-model-4e0323-kserve-56cf4c674f-c48fv\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.804693 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.804643    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-home\") pod \"sub-test-model-4e0323-kserve-56cf4c674f-c48fv\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.804693 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.804665    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-model-cache\") pod \"sub-test-model-4e0323-kserve-56cf4c674f-c48fv\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.806506 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.806486    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-dshm\") pod \"sub-test-model-4e0323-kserve-56cf4c674f-c48fv\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.806756 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.806740    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f27198-ed0b-4261-9194-4a6900a1ff01-tls-certs\") pod \"sub-test-model-4e0323-kserve-56cf4c674f-c48fv\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.813673 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.813651    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phckp\" (UniqueName: \"kubernetes.io/projected/74f27198-ed0b-4261-9194-4a6900a1ff01-kube-api-access-phckp\") pod \"sub-test-model-4e0323-kserve-56cf4c674f-c48fv\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:30.965298 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:30.965169    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:31.236845 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:31.236819    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"]
Sep 11 21:58:31.237929 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:58:31.237903    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74f27198_ed0b_4261_9194_4a6900a1ff01.slice/crio-6692c24155f3d14d31fbc2da0acfbe611ea057d546e7cd05c32c6ab1d3190add WatchSource:0}: Error finding container 6692c24155f3d14d31fbc2da0acfbe611ea057d546e7cd05c32c6ab1d3190add: Status 404 returned error can't find the container with id 6692c24155f3d14d31fbc2da0acfbe611ea057d546e7cd05c32c6ab1d3190add
Sep 11 21:58:31.741938 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:31.741904    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv" event={"ID":"74f27198-ed0b-4261-9194-4a6900a1ff01","Type":"ContainerStarted","Data":"6d75159e72b3d50c7f13fc97987019a8503614204cd6be9c3edd157ac6ee6724"}
Sep 11 21:58:31.741938 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:31.741944    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv" event={"ID":"74f27198-ed0b-4261-9194-4a6900a1ff01","Type":"ContainerStarted","Data":"6692c24155f3d14d31fbc2da0acfbe611ea057d546e7cd05c32c6ab1d3190add"}
Sep 11 21:58:31.758608 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:31.758561    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv" podStartSLOduration=1.758547485 podStartE2EDuration="1.758547485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:58:31.756491872 +0000 UTC m=+1674.725908693" watchObservedRunningTime="2026-09-11 21:58:31.758547485 +0000 UTC m=+1674.727964305"
Sep 11 21:58:37.899461 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:37.899427    2544 scope.go:117] "RemoveContainer" containerID="2d68d2af820237a888f291fb52f13203324988a7113806e75ed180a6d466b6b2"
Sep 11 21:58:40.965909 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:40.965870    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:40.966348 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:40.965921    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:40.978629 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:40.978597    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:58:41.801775 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:58:41.801743    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:59:16.534019 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:16.533981    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"]
Sep 11 21:59:17.393823 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:17.393788    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"]
Sep 11 21:59:31.552811 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.552767    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97" podUID="4d4edd91-13c2-433c-9211-cbeb721d2b7c" containerName="main" containerID="cri-o://3d6b6224f30b99684e130ae8aa6a5b9f29e5d5665b2aa9bb89cd5ee33c875506" gracePeriod=45
Sep 11 21:59:31.803584 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.803521    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:59:31.855145 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.855116    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4edd91-13c2-433c-9211-cbeb721d2b7c-tls-certs\") pod \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") "
Sep 11 21:59:31.855354 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.855161    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-home\") pod \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") "
Sep 11 21:59:31.855354 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.855297    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjkhk\" (UniqueName: \"kubernetes.io/projected/4d4edd91-13c2-433c-9211-cbeb721d2b7c-kube-api-access-cjkhk\") pod \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") "
Sep 11 21:59:31.855354 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.855328    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-tmp-dir\") pod \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") "
Sep 11 21:59:31.855541 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.855382    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-model-cache\") pod \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") "
Sep 11 21:59:31.855541 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.855417    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-dshm\") pod \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\" (UID: \"4d4edd91-13c2-433c-9211-cbeb721d2b7c\") "
Sep 11 21:59:31.855541 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.855423    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-home" (OuterVolumeSpecName: "home") pod "4d4edd91-13c2-433c-9211-cbeb721d2b7c" (UID: "4d4edd91-13c2-433c-9211-cbeb721d2b7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:59:31.855711 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.855588    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d4edd91-13c2-433c-9211-cbeb721d2b7c" (UID: "4d4edd91-13c2-433c-9211-cbeb721d2b7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:59:31.855711 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.855646    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d4edd91-13c2-433c-9211-cbeb721d2b7c" (UID: "4d4edd91-13c2-433c-9211-cbeb721d2b7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:59:31.855711 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.855706    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:59:31.855873 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.855718    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:59:31.857380 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.857350    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-dshm" (OuterVolumeSpecName: "dshm") pod "4d4edd91-13c2-433c-9211-cbeb721d2b7c" (UID: "4d4edd91-13c2-433c-9211-cbeb721d2b7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:59:31.857530 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.857418    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4edd91-13c2-433c-9211-cbeb721d2b7c-kube-api-access-cjkhk" (OuterVolumeSpecName: "kube-api-access-cjkhk") pod "4d4edd91-13c2-433c-9211-cbeb721d2b7c" (UID: "4d4edd91-13c2-433c-9211-cbeb721d2b7c"). InnerVolumeSpecName "kube-api-access-cjkhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:59:31.857530 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.857470    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d4edd91-13c2-433c-9211-cbeb721d2b7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d4edd91-13c2-433c-9211-cbeb721d2b7c" (UID: "4d4edd91-13c2-433c-9211-cbeb721d2b7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:59:31.956388 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.956352    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:59:31.956388 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.956382    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4edd91-13c2-433c-9211-cbeb721d2b7c-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:59:31.956388 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.956393    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjkhk\" (UniqueName: \"kubernetes.io/projected/4d4edd91-13c2-433c-9211-cbeb721d2b7c-kube-api-access-cjkhk\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:59:31.956621 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:31.956402    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4edd91-13c2-433c-9211-cbeb721d2b7c-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:59:32.030406 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.030373    2544 generic.go:356] "Generic (PLEG): container finished" podID="4d4edd91-13c2-433c-9211-cbeb721d2b7c" containerID="3d6b6224f30b99684e130ae8aa6a5b9f29e5d5665b2aa9bb89cd5ee33c875506" exitCode=0
Sep 11 21:59:32.030588 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.030416    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"
Sep 11 21:59:32.030588 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.030467    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97" event={"ID":"4d4edd91-13c2-433c-9211-cbeb721d2b7c","Type":"ContainerDied","Data":"3d6b6224f30b99684e130ae8aa6a5b9f29e5d5665b2aa9bb89cd5ee33c875506"}
Sep 11 21:59:32.030588 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.030516    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97" event={"ID":"4d4edd91-13c2-433c-9211-cbeb721d2b7c","Type":"ContainerDied","Data":"00e7b1908a9d12f8cf2376acb215ea7bff6164444e15fcc446a53b0955b007de"}
Sep 11 21:59:32.030588 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.030540    2544 scope.go:117] "RemoveContainer" containerID="3d6b6224f30b99684e130ae8aa6a5b9f29e5d5665b2aa9bb89cd5ee33c875506"
Sep 11 21:59:32.041284 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.041263    2544 scope.go:117] "RemoveContainer" containerID="3d6b6224f30b99684e130ae8aa6a5b9f29e5d5665b2aa9bb89cd5ee33c875506"
Sep 11 21:59:32.041837 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:59:32.041812    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d6b6224f30b99684e130ae8aa6a5b9f29e5d5665b2aa9bb89cd5ee33c875506\": container with ID starting with 3d6b6224f30b99684e130ae8aa6a5b9f29e5d5665b2aa9bb89cd5ee33c875506 not found: ID does not exist" containerID="3d6b6224f30b99684e130ae8aa6a5b9f29e5d5665b2aa9bb89cd5ee33c875506"
Sep 11 21:59:32.041924 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.041848    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6b6224f30b99684e130ae8aa6a5b9f29e5d5665b2aa9bb89cd5ee33c875506"} err="failed to get container status \"3d6b6224f30b99684e130ae8aa6a5b9f29e5d5665b2aa9bb89cd5ee33c875506\": rpc error: code = NotFound desc = could not find container \"3d6b6224f30b99684e130ae8aa6a5b9f29e5d5665b2aa9bb89cd5ee33c875506\": container with ID starting with 3d6b6224f30b99684e130ae8aa6a5b9f29e5d5665b2aa9bb89cd5ee33c875506 not found: ID does not exist"
Sep 11 21:59:32.058370 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.058314    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"]
Sep 11 21:59:32.064730 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.064709    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7d7821/sub-test-model-7d7821-kserve-68645548f7-5ds97"]
Sep 11 21:59:32.412860 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.412770    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv" podUID="74f27198-ed0b-4261-9194-4a6900a1ff01" containerName="main" containerID="cri-o://6d75159e72b3d50c7f13fc97987019a8503614204cd6be9c3edd157ac6ee6724" gracePeriod=45
Sep 11 21:59:32.658171 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.658145    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:59:32.761661 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.761635    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phckp\" (UniqueName: \"kubernetes.io/projected/74f27198-ed0b-4261-9194-4a6900a1ff01-kube-api-access-phckp\") pod \"74f27198-ed0b-4261-9194-4a6900a1ff01\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") "
Sep 11 21:59:32.761821 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.761681    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-dshm\") pod \"74f27198-ed0b-4261-9194-4a6900a1ff01\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") "
Sep 11 21:59:32.761821 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.761705    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-tmp-dir\") pod \"74f27198-ed0b-4261-9194-4a6900a1ff01\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") "
Sep 11 21:59:32.761821 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.761748    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f27198-ed0b-4261-9194-4a6900a1ff01-tls-certs\") pod \"74f27198-ed0b-4261-9194-4a6900a1ff01\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") "
Sep 11 21:59:32.761821 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.761789    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-model-cache\") pod \"74f27198-ed0b-4261-9194-4a6900a1ff01\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") "
Sep 11 21:59:32.761821 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.761810    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-home\") pod \"74f27198-ed0b-4261-9194-4a6900a1ff01\" (UID: \"74f27198-ed0b-4261-9194-4a6900a1ff01\") "
Sep 11 21:59:32.762088 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.762025    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-model-cache" (OuterVolumeSpecName: "model-cache") pod "74f27198-ed0b-4261-9194-4a6900a1ff01" (UID: "74f27198-ed0b-4261-9194-4a6900a1ff01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:59:32.762088 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.762044    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "74f27198-ed0b-4261-9194-4a6900a1ff01" (UID: "74f27198-ed0b-4261-9194-4a6900a1ff01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:59:32.762169 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.762150    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-home" (OuterVolumeSpecName: "home") pod "74f27198-ed0b-4261-9194-4a6900a1ff01" (UID: "74f27198-ed0b-4261-9194-4a6900a1ff01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:59:32.763827 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.763801    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f27198-ed0b-4261-9194-4a6900a1ff01-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "74f27198-ed0b-4261-9194-4a6900a1ff01" (UID: "74f27198-ed0b-4261-9194-4a6900a1ff01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:59:32.763934 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.763834    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-dshm" (OuterVolumeSpecName: "dshm") pod "74f27198-ed0b-4261-9194-4a6900a1ff01" (UID: "74f27198-ed0b-4261-9194-4a6900a1ff01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:59:32.763934 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.763896    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f27198-ed0b-4261-9194-4a6900a1ff01-kube-api-access-phckp" (OuterVolumeSpecName: "kube-api-access-phckp") pod "74f27198-ed0b-4261-9194-4a6900a1ff01" (UID: "74f27198-ed0b-4261-9194-4a6900a1ff01"). InnerVolumeSpecName "kube-api-access-phckp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:59:32.863052 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.863024    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:59:32.863052 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.863049    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:59:32.863052 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.863058    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f27198-ed0b-4261-9194-4a6900a1ff01-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:59:32.863269 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.863067    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:59:32.863269 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.863074    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74f27198-ed0b-4261-9194-4a6900a1ff01-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:59:32.863269 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:32.863083    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phckp\" (UniqueName: \"kubernetes.io/projected/74f27198-ed0b-4261-9194-4a6900a1ff01-kube-api-access-phckp\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 21:59:33.036322 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:33.036241    2544 generic.go:356] "Generic (PLEG): container finished" podID="74f27198-ed0b-4261-9194-4a6900a1ff01" containerID="6d75159e72b3d50c7f13fc97987019a8503614204cd6be9c3edd157ac6ee6724" exitCode=0
Sep 11 21:59:33.036322 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:33.036293    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"
Sep 11 21:59:33.036498 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:33.036333    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv" event={"ID":"74f27198-ed0b-4261-9194-4a6900a1ff01","Type":"ContainerDied","Data":"6d75159e72b3d50c7f13fc97987019a8503614204cd6be9c3edd157ac6ee6724"}
Sep 11 21:59:33.036498 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:33.036383    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv" event={"ID":"74f27198-ed0b-4261-9194-4a6900a1ff01","Type":"ContainerDied","Data":"6692c24155f3d14d31fbc2da0acfbe611ea057d546e7cd05c32c6ab1d3190add"}
Sep 11 21:59:33.036498 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:33.036403    2544 scope.go:117] "RemoveContainer" containerID="6d75159e72b3d50c7f13fc97987019a8503614204cd6be9c3edd157ac6ee6724"
Sep 11 21:59:33.045124 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:33.045107    2544 scope.go:117] "RemoveContainer" containerID="6d75159e72b3d50c7f13fc97987019a8503614204cd6be9c3edd157ac6ee6724"
Sep 11 21:59:33.045406 ip-10-0-142-211 kubenswrapper[2544]: E0911 21:59:33.045384    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d75159e72b3d50c7f13fc97987019a8503614204cd6be9c3edd157ac6ee6724\": container with ID starting with 6d75159e72b3d50c7f13fc97987019a8503614204cd6be9c3edd157ac6ee6724 not found: ID does not exist" containerID="6d75159e72b3d50c7f13fc97987019a8503614204cd6be9c3edd157ac6ee6724"
Sep 11 21:59:33.045481 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:33.045412    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d75159e72b3d50c7f13fc97987019a8503614204cd6be9c3edd157ac6ee6724"} err="failed to get container status \"6d75159e72b3d50c7f13fc97987019a8503614204cd6be9c3edd157ac6ee6724\": rpc error: code = NotFound desc = could not find container \"6d75159e72b3d50c7f13fc97987019a8503614204cd6be9c3edd157ac6ee6724\": container with ID starting with 6d75159e72b3d50c7f13fc97987019a8503614204cd6be9c3edd157ac6ee6724 not found: ID does not exist"
Sep 11 21:59:33.057277 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:33.057254    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"]
Sep 11 21:59:33.060839 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:33.060820    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e0323/sub-test-model-4e0323-kserve-56cf4c674f-c48fv"]
Sep 11 21:59:33.592701 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:33.592671    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d4edd91-13c2-433c-9211-cbeb721d2b7c" path="/var/lib/kubelet/pods/4d4edd91-13c2-433c-9211-cbeb721d2b7c/volumes"
Sep 11 21:59:33.593051 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:33.593038    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74f27198-ed0b-4261-9194-4a6900a1ff01" path="/var/lib/kubelet/pods/74f27198-ed0b-4261-9194-4a6900a1ff01/volumes"
Sep 11 21:59:34.448866 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.448829    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"]
Sep 11 21:59:34.449273 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.449144    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d4edd91-13c2-433c-9211-cbeb721d2b7c" containerName="main"
Sep 11 21:59:34.449273 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.449155    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d4edd91-13c2-433c-9211-cbeb721d2b7c" containerName="main"
Sep 11 21:59:34.449273 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.449164    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74f27198-ed0b-4261-9194-4a6900a1ff01" containerName="main"
Sep 11 21:59:34.449273 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.449170    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="74f27198-ed0b-4261-9194-4a6900a1ff01" containerName="main"
Sep 11 21:59:34.449273 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.449234    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d4edd91-13c2-433c-9211-cbeb721d2b7c" containerName="main"
Sep 11 21:59:34.449273 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.449243    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="74f27198-ed0b-4261-9194-4a6900a1ff01" containerName="main"
Sep 11 21:59:34.453510 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.453495    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.456039 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.456010    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7d7821\"/\"kube-root-ca.crt\""
Sep 11 21:59:34.457062 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.457039    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7d7821\"/\"openshift-service-ca.crt\""
Sep 11 21:59:34.457238 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.457050    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7d7821\"/\"rate-test-model-7d7821-kserve-self-signed-certs\""
Sep 11 21:59:34.457238 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.457106    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7d7821\"/\"default-dockercfg-bvvhd\""
Sep 11 21:59:34.459995 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.459972    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"]
Sep 11 21:59:34.576791 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.576761    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf2wt\" (UniqueName: \"kubernetes.io/projected/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-kube-api-access-mf2wt\") pod \"rate-test-model-7d7821-kserve-788d98fcb7-qclfx\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.576946 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.576812    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-tls-certs\") pod \"rate-test-model-7d7821-kserve-788d98fcb7-qclfx\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.576946 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.576862    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-model-cache\") pod \"rate-test-model-7d7821-kserve-788d98fcb7-qclfx\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.576946 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.576881    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-tmp-dir\") pod \"rate-test-model-7d7821-kserve-788d98fcb7-qclfx\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.577061 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.576945    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-dshm\") pod \"rate-test-model-7d7821-kserve-788d98fcb7-qclfx\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.577061 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.576988    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-home\") pod \"rate-test-model-7d7821-kserve-788d98fcb7-qclfx\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.677709 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.677678    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-model-cache\") pod \"rate-test-model-7d7821-kserve-788d98fcb7-qclfx\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.677896 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.677719    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-tmp-dir\") pod \"rate-test-model-7d7821-kserve-788d98fcb7-qclfx\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.677896 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.677752    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-dshm\") pod \"rate-test-model-7d7821-kserve-788d98fcb7-qclfx\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.677896 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.677834    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-home\") pod \"rate-test-model-7d7821-kserve-788d98fcb7-qclfx\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.677896 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.677884    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf2wt\" (UniqueName: \"kubernetes.io/projected/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-kube-api-access-mf2wt\") pod \"rate-test-model-7d7821-kserve-788d98fcb7-qclfx\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.678125 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.677952    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-tls-certs\") pod \"rate-test-model-7d7821-kserve-788d98fcb7-qclfx\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.678125 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.678108    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-model-cache\") pod \"rate-test-model-7d7821-kserve-788d98fcb7-qclfx\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.678280 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.678159    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-tmp-dir\") pod \"rate-test-model-7d7821-kserve-788d98fcb7-qclfx\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.678423 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.678407    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-home\") pod \"rate-test-model-7d7821-kserve-788d98fcb7-qclfx\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.680121 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.680095    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-dshm\") pod \"rate-test-model-7d7821-kserve-788d98fcb7-qclfx\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.680433 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.680413    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-tls-certs\") pod \"rate-test-model-7d7821-kserve-788d98fcb7-qclfx\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.685966 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.685947    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf2wt\" (UniqueName: \"kubernetes.io/projected/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-kube-api-access-mf2wt\") pod \"rate-test-model-7d7821-kserve-788d98fcb7-qclfx\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") " pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:34.763101 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:34.763066    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:35.033390 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:35.033297    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"]
Sep 11 21:59:35.037372 ip-10-0-142-211 kubenswrapper[2544]: W0911 21:59:35.037341    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f7ee29c_af7b_4ffb_a3af_f70628aa950a.slice/crio-d69a0ef863988e4aff22fd939d0bac83a65700d8bc14047fda264583a076c328 WatchSource:0}: Error finding container d69a0ef863988e4aff22fd939d0bac83a65700d8bc14047fda264583a076c328: Status 404 returned error can't find the container with id d69a0ef863988e4aff22fd939d0bac83a65700d8bc14047fda264583a076c328
Sep 11 21:59:35.049522 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:35.049497    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx" event={"ID":"4f7ee29c-af7b-4ffb-a3af-f70628aa950a","Type":"ContainerStarted","Data":"d69a0ef863988e4aff22fd939d0bac83a65700d8bc14047fda264583a076c328"}
Sep 11 21:59:36.056001 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:36.055965    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx" event={"ID":"4f7ee29c-af7b-4ffb-a3af-f70628aa950a","Type":"ContainerStarted","Data":"23327dc4ce60d13c37d0c988a35d3614146e7a56e493d449dd2f74405889d13b"}
Sep 11 21:59:36.073687 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:36.073640    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx" podStartSLOduration=2.073624232 podStartE2EDuration="2.073624232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:59:36.071926142 +0000 UTC m=+1739.041342962" watchObservedRunningTime="2026-09-11 21:59:36.073624232 +0000 UTC m=+1739.043041055"
Sep 11 21:59:37.921348 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:37.921310    2544 scope.go:117] "RemoveContainer" containerID="f84622996f3e1081a4d94b2ff219091ddc913446bcd790b0ec92b8539588c38c"
Sep 11 21:59:44.763954 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:44.763920    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:44.764389 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:44.763988    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:44.776504 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:44.776478    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 21:59:45.109279 ip-10-0-142-211 kubenswrapper[2544]: I0911 21:59:45.109192    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 22:00:00.137083 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:00.137048    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7d7821-29819400-4wf67"]
Sep 11 22:00:00.140424 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:00.140407    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7d7821-29819400-4wf67"
Sep 11 22:00:00.142931 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:00.142912    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 22:00:00.149864 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:00.149693    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7d7821-29819400-4wf67"
Sep 11 22:00:00.152235 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:00.152191    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7d7821-29819400-4wf67"]
Sep 11 22:00:00.420090 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:00.420019    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7d7821-29819400-4wf67"]
Sep 11 22:00:00.423358 ip-10-0-142-211 kubenswrapper[2544]: W0911 22:00:00.423332    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb2d38d_e320_4dae_8f1d_0100cc3624a9.slice/crio-a21b4ea0177516f766b4d5523ac71700c7c530051e24739e4b7bf809b8a6c083 WatchSource:0}: Error finding container a21b4ea0177516f766b4d5523ac71700c7c530051e24739e4b7bf809b8a6c083: Status 404 returned error can't find the container with id a21b4ea0177516f766b4d5523ac71700c7c530051e24739e4b7bf809b8a6c083
Sep 11 22:00:01.172995 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:01.172913    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7d7821-29819400-4wf67" event={"ID":"dcb2d38d-e320-4dae-8f1d-0100cc3624a9","Type":"ContainerStarted","Data":"d6426f321c9b3062717547235fe828afe099027dc99861cedefe4b50ab735a6b"}
Sep 11 22:00:01.172995 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:01.172963    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7d7821-29819400-4wf67" event={"ID":"dcb2d38d-e320-4dae-8f1d-0100cc3624a9","Type":"ContainerStarted","Data":"a21b4ea0177516f766b4d5523ac71700c7c530051e24739e4b7bf809b8a6c083"}
Sep 11 22:00:01.188354 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:01.188312    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7d7821-29819400-4wf67" podStartSLOduration=1.188297796 podStartE2EDuration="1.188297796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:00:01.186739364 +0000 UTC m=+1764.156156184" watchObservedRunningTime="2026-09-11 22:00:01.188297796 +0000 UTC m=+1764.157714615"
Sep 11 22:00:02.179154 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:02.179119    2544 generic.go:356] "Generic (PLEG): container finished" podID="dcb2d38d-e320-4dae-8f1d-0100cc3624a9" containerID="d6426f321c9b3062717547235fe828afe099027dc99861cedefe4b50ab735a6b" exitCode=0
Sep 11 22:00:02.179698 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:02.179183    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7d7821-29819400-4wf67" event={"ID":"dcb2d38d-e320-4dae-8f1d-0100cc3624a9","Type":"ContainerDied","Data":"d6426f321c9b3062717547235fe828afe099027dc99861cedefe4b50ab735a6b"}
Sep 11 22:00:03.315181 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:03.315153    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7d7821-29819400-4wf67"
Sep 11 22:00:04.190967 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:04.190937    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7d7821-29819400-4wf67"
Sep 11 22:00:04.191157 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:04.190937    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7d7821-29819400-4wf67" event={"ID":"dcb2d38d-e320-4dae-8f1d-0100cc3624a9","Type":"ContainerDied","Data":"a21b4ea0177516f766b4d5523ac71700c7c530051e24739e4b7bf809b8a6c083"}
Sep 11 22:00:04.191157 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:04.191068    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a21b4ea0177516f766b4d5523ac71700c7c530051e24739e4b7bf809b8a6c083"
Sep 11 22:00:07.872064 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:07.872028    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"]
Sep 11 22:00:07.872549 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:07.872532    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcb2d38d-e320-4dae-8f1d-0100cc3624a9" containerName="cleanup"
Sep 11 22:00:07.872593 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:07.872552    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcb2d38d-e320-4dae-8f1d-0100cc3624a9" containerName="cleanup"
Sep 11 22:00:07.872659 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:07.872649    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcb2d38d-e320-4dae-8f1d-0100cc3624a9" containerName="cleanup"
Sep 11 22:00:07.875853 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:07.875832    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:07.878425 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:07.878404    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4e0323\"/\"rate-test-model-4e0323-kserve-self-signed-certs\""
Sep 11 22:00:07.878554 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:07.878516    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4e0323\"/\"openshift-service-ca.crt\""
Sep 11 22:00:07.879290 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:07.879266    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4e0323\"/\"default-dockercfg-lp4bk\""
Sep 11 22:00:07.879413 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:07.879331    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4e0323\"/\"kube-root-ca.crt\""
Sep 11 22:00:07.883365 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:07.883193    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"]
Sep 11 22:00:07.967985 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:07.967956    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-home\") pod \"rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:07.968141 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:07.967991    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-dshm\") pod \"rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:07.968141 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:07.968020    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfk9r\" (UniqueName: \"kubernetes.io/projected/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-kube-api-access-qfk9r\") pod \"rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:07.968141 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:07.968100    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-tmp-dir\") pod \"rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:07.968141 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:07.968124    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-tls-certs\") pod \"rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:07.968301 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:07.968163    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-model-cache\") pod \"rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:08.069508 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:08.069475    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-home\") pod \"rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:08.069681 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:08.069526    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-dshm\") pod \"rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:08.069681 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:08.069578    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfk9r\" (UniqueName: \"kubernetes.io/projected/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-kube-api-access-qfk9r\") pod \"rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:08.069681 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:08.069630    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-tmp-dir\") pod \"rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:08.069681 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:08.069652    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-tls-certs\") pod \"rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:08.069902 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:08.069880    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-model-cache\") pod \"rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:08.069963 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:08.069933    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-home\") pod \"rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:08.070171 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:08.070144    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-tmp-dir\") pod \"rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:08.070325 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:08.070299    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-model-cache\") pod \"rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:08.072136 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:08.072105    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-dshm\") pod \"rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:08.072353 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:08.072335    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-tls-certs\") pod \"rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:08.077845 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:08.077823    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfk9r\" (UniqueName: \"kubernetes.io/projected/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-kube-api-access-qfk9r\") pod \"rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:08.186783 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:08.186693    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:08.456593 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:08.456569    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"]
Sep 11 22:00:08.459036 ip-10-0-142-211 kubenswrapper[2544]: W0911 22:00:08.459009    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b201b68_a1dd_4ded_bda0_2a1f40c976ef.slice/crio-503be86822dc4b69c18b329eefa2b55d472c47af2325fc55c1e3744879fba912 WatchSource:0}: Error finding container 503be86822dc4b69c18b329eefa2b55d472c47af2325fc55c1e3744879fba912: Status 404 returned error can't find the container with id 503be86822dc4b69c18b329eefa2b55d472c47af2325fc55c1e3744879fba912
Sep 11 22:00:09.217031 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:09.216994    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr" event={"ID":"4b201b68-a1dd-4ded-bda0-2a1f40c976ef","Type":"ContainerStarted","Data":"5d885a84c7e99f0389aacfccfcc8dbbd1dfe101680f19514d30970fcd0e20ddc"}
Sep 11 22:00:09.217031 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:09.217038    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr" event={"ID":"4b201b68-a1dd-4ded-bda0-2a1f40c976ef","Type":"ContainerStarted","Data":"503be86822dc4b69c18b329eefa2b55d472c47af2325fc55c1e3744879fba912"}
Sep 11 22:00:09.233195 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:09.233148    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr" podStartSLOduration=2.233132424 podStartE2EDuration="2.233132424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:00:09.231357524 +0000 UTC m=+1772.200774345" watchObservedRunningTime="2026-09-11 22:00:09.233132424 +0000 UTC m=+1772.202549244"
Sep 11 22:00:18.187073 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:18.187032    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:18.187511 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:18.187085    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:18.199890 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:18.199868    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:00:18.270924 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:00:18.270890    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:01:17.256782 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:17.256746    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"]
Sep 11 22:01:18.246346 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:18.246312    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"]
Sep 11 22:01:32.275568 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.275522    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx" podUID="4f7ee29c-af7b-4ffb-a3af-f70628aa950a" containerName="main" containerID="cri-o://23327dc4ce60d13c37d0c988a35d3614146e7a56e493d449dd2f74405889d13b" gracePeriod=45
Sep 11 22:01:32.534437 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.534374    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 22:01:32.612838 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.612796    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-model-cache\") pod \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") "
Sep 11 22:01:32.613041 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.612870    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-tls-certs\") pod \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") "
Sep 11 22:01:32.613041 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.612904    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-dshm\") pod \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") "
Sep 11 22:01:32.613166 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.613041    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mf2wt\" (UniqueName: \"kubernetes.io/projected/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-kube-api-access-mf2wt\") pod \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") "
Sep 11 22:01:32.613166 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.613098    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-home\") pod \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") "
Sep 11 22:01:32.613166 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.613121    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f7ee29c-af7b-4ffb-a3af-f70628aa950a" (UID: "4f7ee29c-af7b-4ffb-a3af-f70628aa950a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:01:32.613380 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.613166    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-tmp-dir\") pod \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\" (UID: \"4f7ee29c-af7b-4ffb-a3af-f70628aa950a\") "
Sep 11 22:01:32.613434 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.613388    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f7ee29c-af7b-4ffb-a3af-f70628aa950a" (UID: "4f7ee29c-af7b-4ffb-a3af-f70628aa950a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:01:32.613483 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.613422    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-home" (OuterVolumeSpecName: "home") pod "4f7ee29c-af7b-4ffb-a3af-f70628aa950a" (UID: "4f7ee29c-af7b-4ffb-a3af-f70628aa950a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:01:32.613544 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.613531    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:01:32.613601 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.613549    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:01:32.613601 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.613563    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:01:32.615032 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.615009    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f7ee29c-af7b-4ffb-a3af-f70628aa950a" (UID: "4f7ee29c-af7b-4ffb-a3af-f70628aa950a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:01:32.615032 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.615014    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-dshm" (OuterVolumeSpecName: "dshm") pod "4f7ee29c-af7b-4ffb-a3af-f70628aa950a" (UID: "4f7ee29c-af7b-4ffb-a3af-f70628aa950a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:01:32.615164 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.615115    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-kube-api-access-mf2wt" (OuterVolumeSpecName: "kube-api-access-mf2wt") pod "4f7ee29c-af7b-4ffb-a3af-f70628aa950a" (UID: "4f7ee29c-af7b-4ffb-a3af-f70628aa950a"). InnerVolumeSpecName "kube-api-access-mf2wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:01:32.625734 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.625703    2544 generic.go:356] "Generic (PLEG): container finished" podID="4f7ee29c-af7b-4ffb-a3af-f70628aa950a" containerID="23327dc4ce60d13c37d0c988a35d3614146e7a56e493d449dd2f74405889d13b" exitCode=0
Sep 11 22:01:32.625843 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.625750    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"
Sep 11 22:01:32.625843 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.625787    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx" event={"ID":"4f7ee29c-af7b-4ffb-a3af-f70628aa950a","Type":"ContainerDied","Data":"23327dc4ce60d13c37d0c988a35d3614146e7a56e493d449dd2f74405889d13b"}
Sep 11 22:01:32.625843 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.625835    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx" event={"ID":"4f7ee29c-af7b-4ffb-a3af-f70628aa950a","Type":"ContainerDied","Data":"d69a0ef863988e4aff22fd939d0bac83a65700d8bc14047fda264583a076c328"}
Sep 11 22:01:32.625947 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.625853    2544 scope.go:117] "RemoveContainer" containerID="23327dc4ce60d13c37d0c988a35d3614146e7a56e493d449dd2f74405889d13b"
Sep 11 22:01:32.634892 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.634872    2544 scope.go:117] "RemoveContainer" containerID="23327dc4ce60d13c37d0c988a35d3614146e7a56e493d449dd2f74405889d13b"
Sep 11 22:01:32.635182 ip-10-0-142-211 kubenswrapper[2544]: E0911 22:01:32.635164    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23327dc4ce60d13c37d0c988a35d3614146e7a56e493d449dd2f74405889d13b\": container with ID starting with 23327dc4ce60d13c37d0c988a35d3614146e7a56e493d449dd2f74405889d13b not found: ID does not exist" containerID="23327dc4ce60d13c37d0c988a35d3614146e7a56e493d449dd2f74405889d13b"
Sep 11 22:01:32.635260 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.635191    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23327dc4ce60d13c37d0c988a35d3614146e7a56e493d449dd2f74405889d13b"} err="failed to get container status \"23327dc4ce60d13c37d0c988a35d3614146e7a56e493d449dd2f74405889d13b\": rpc error: code = NotFound desc = could not find container \"23327dc4ce60d13c37d0c988a35d3614146e7a56e493d449dd2f74405889d13b\": container with ID starting with 23327dc4ce60d13c37d0c988a35d3614146e7a56e493d449dd2f74405889d13b not found: ID does not exist"
Sep 11 22:01:32.645889 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.645859    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"]
Sep 11 22:01:32.649958 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.649929    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7d7821/rate-test-model-7d7821-kserve-788d98fcb7-qclfx"]
Sep 11 22:01:32.714474 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.714440    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:01:32.714474 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.714466    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:01:32.714474 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:32.714477    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf2wt\" (UniqueName: \"kubernetes.io/projected/4f7ee29c-af7b-4ffb-a3af-f70628aa950a-kube-api-access-mf2wt\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:01:33.265950 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.265904    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr" podUID="4b201b68-a1dd-4ded-bda0-2a1f40c976ef" containerName="main" containerID="cri-o://5d885a84c7e99f0389aacfccfcc8dbbd1dfe101680f19514d30970fcd0e20ddc" gracePeriod=45
Sep 11 22:01:33.531254 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.531206    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:01:33.591621 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.591578    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f7ee29c-af7b-4ffb-a3af-f70628aa950a" path="/var/lib/kubelet/pods/4f7ee29c-af7b-4ffb-a3af-f70628aa950a/volumes"
Sep 11 22:01:33.622742 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.622704    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfk9r\" (UniqueName: \"kubernetes.io/projected/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-kube-api-access-qfk9r\") pod \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") "
Sep 11 22:01:33.622934 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.622754    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-dshm\") pod \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") "
Sep 11 22:01:33.622934 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.622776    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-home\") pod \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") "
Sep 11 22:01:33.622934 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.622848    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-tmp-dir\") pod \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") "
Sep 11 22:01:33.622934 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.622886    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-model-cache\") pod \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") "
Sep 11 22:01:33.622934 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.622909    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-tls-certs\") pod \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\" (UID: \"4b201b68-a1dd-4ded-bda0-2a1f40c976ef\") "
Sep 11 22:01:33.623200 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.623118    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-home" (OuterVolumeSpecName: "home") pod "4b201b68-a1dd-4ded-bda0-2a1f40c976ef" (UID: "4b201b68-a1dd-4ded-bda0-2a1f40c976ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:01:33.623200 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.623125    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b201b68-a1dd-4ded-bda0-2a1f40c976ef" (UID: "4b201b68-a1dd-4ded-bda0-2a1f40c976ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:01:33.623200 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.623156    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b201b68-a1dd-4ded-bda0-2a1f40c976ef" (UID: "4b201b68-a1dd-4ded-bda0-2a1f40c976ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:01:33.624993 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.624967    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-kube-api-access-qfk9r" (OuterVolumeSpecName: "kube-api-access-qfk9r") pod "4b201b68-a1dd-4ded-bda0-2a1f40c976ef" (UID: "4b201b68-a1dd-4ded-bda0-2a1f40c976ef"). InnerVolumeSpecName "kube-api-access-qfk9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:01:33.625128 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.625064    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b201b68-a1dd-4ded-bda0-2a1f40c976ef" (UID: "4b201b68-a1dd-4ded-bda0-2a1f40c976ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:01:33.625128 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.625071    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-dshm" (OuterVolumeSpecName: "dshm") pod "4b201b68-a1dd-4ded-bda0-2a1f40c976ef" (UID: "4b201b68-a1dd-4ded-bda0-2a1f40c976ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:01:33.632143 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.632114    2544 generic.go:356] "Generic (PLEG): container finished" podID="4b201b68-a1dd-4ded-bda0-2a1f40c976ef" containerID="5d885a84c7e99f0389aacfccfcc8dbbd1dfe101680f19514d30970fcd0e20ddc" exitCode=0
Sep 11 22:01:33.632291 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.632162    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"
Sep 11 22:01:33.632291 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.632201    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr" event={"ID":"4b201b68-a1dd-4ded-bda0-2a1f40c976ef","Type":"ContainerDied","Data":"5d885a84c7e99f0389aacfccfcc8dbbd1dfe101680f19514d30970fcd0e20ddc"}
Sep 11 22:01:33.632291 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.632267    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr" event={"ID":"4b201b68-a1dd-4ded-bda0-2a1f40c976ef","Type":"ContainerDied","Data":"503be86822dc4b69c18b329eefa2b55d472c47af2325fc55c1e3744879fba912"}
Sep 11 22:01:33.632291 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.632290    2544 scope.go:117] "RemoveContainer" containerID="5d885a84c7e99f0389aacfccfcc8dbbd1dfe101680f19514d30970fcd0e20ddc"
Sep 11 22:01:33.643010 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.642990    2544 scope.go:117] "RemoveContainer" containerID="5d885a84c7e99f0389aacfccfcc8dbbd1dfe101680f19514d30970fcd0e20ddc"
Sep 11 22:01:33.643346 ip-10-0-142-211 kubenswrapper[2544]: E0911 22:01:33.643325    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d885a84c7e99f0389aacfccfcc8dbbd1dfe101680f19514d30970fcd0e20ddc\": container with ID starting with 5d885a84c7e99f0389aacfccfcc8dbbd1dfe101680f19514d30970fcd0e20ddc not found: ID does not exist" containerID="5d885a84c7e99f0389aacfccfcc8dbbd1dfe101680f19514d30970fcd0e20ddc"
Sep 11 22:01:33.643389 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.643354    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d885a84c7e99f0389aacfccfcc8dbbd1dfe101680f19514d30970fcd0e20ddc"} err="failed to get container status \"5d885a84c7e99f0389aacfccfcc8dbbd1dfe101680f19514d30970fcd0e20ddc\": rpc error: code = NotFound desc = could not find container \"5d885a84c7e99f0389aacfccfcc8dbbd1dfe101680f19514d30970fcd0e20ddc\": container with ID starting with 5d885a84c7e99f0389aacfccfcc8dbbd1dfe101680f19514d30970fcd0e20ddc not found: ID does not exist"
Sep 11 22:01:33.654316 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.654285    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"]
Sep 11 22:01:33.657786 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.657762    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e0323/rate-test-model-4e0323-kserve-654bc5dcc5-2c7hr"]
Sep 11 22:01:33.724348 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.724313    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:01:33.724348 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.724342    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:01:33.724348 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.724351    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:01:33.724579 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.724359    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfk9r\" (UniqueName: \"kubernetes.io/projected/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-kube-api-access-qfk9r\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:01:33.724579 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.724370    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:01:33.724579 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:33.724381    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b201b68-a1dd-4ded-bda0-2a1f40c976ef-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:01:35.592245 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:35.592193    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b201b68-a1dd-4ded-bda0-2a1f40c976ef" path="/var/lib/kubelet/pods/4b201b68-a1dd-4ded-bda0-2a1f40c976ef/volumes"
Sep 11 22:01:47.838461 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.838426    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"]
Sep 11 22:01:47.838854 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.838763    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b201b68-a1dd-4ded-bda0-2a1f40c976ef" containerName="main"
Sep 11 22:01:47.838854 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.838777    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b201b68-a1dd-4ded-bda0-2a1f40c976ef" containerName="main"
Sep 11 22:01:47.838854 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.838801    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f7ee29c-af7b-4ffb-a3af-f70628aa950a" containerName="main"
Sep 11 22:01:47.838854 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.838807    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f7ee29c-af7b-4ffb-a3af-f70628aa950a" containerName="main"
Sep 11 22:01:47.839012 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.838874    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b201b68-a1dd-4ded-bda0-2a1f40c976ef" containerName="main"
Sep 11 22:01:47.839012 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.838885    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f7ee29c-af7b-4ffb-a3af-f70628aa950a" containerName="main"
Sep 11 22:01:47.841862 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.841843    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.844359 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.844330    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-077193-openshift-default-dockercfg-8522k\""
Sep 11 22:01:47.850133 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.850111    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-data\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.850245 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.850141    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-workload-certs\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.850245 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.850157    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-token\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.850333 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.850236    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl9zf\" (UniqueName: \"kubernetes.io/projected/7bc05684-948b-414b-9a71-ba8f47eaba93-kube-api-access-hl9zf\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.850333 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.850289    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-credential-socket\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.850434 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.850341    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-envoy\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.850434 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.850413    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-workload-socket\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.850532 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.850450    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-podinfo\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.850532 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.850496    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bc05684-948b-414b-9a71-ba8f47eaba93-istiod-ca-cert\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.853520 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.853498    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"]
Sep 11 22:01:47.951910 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.951868    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-workload-socket\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.951910 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.951905    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-podinfo\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.952202 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.951930    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bc05684-948b-414b-9a71-ba8f47eaba93-istiod-ca-cert\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.952202 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.951962    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-data\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.952202 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.951990    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-workload-certs\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.952202 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.952015    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-token\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.952202 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.952059    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl9zf\" (UniqueName: \"kubernetes.io/projected/7bc05684-948b-414b-9a71-ba8f47eaba93-kube-api-access-hl9zf\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.952202 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.952083    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-credential-socket\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.952202 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.952116    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-envoy\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.952750 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.952516    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-workload-certs\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.952750 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.952626    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-data\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.952986 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.952952    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bc05684-948b-414b-9a71-ba8f47eaba93-istiod-ca-cert\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.953063 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.952959    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-workload-socket\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.953202 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.953183    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-credential-socket\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.954515 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.954490    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-envoy\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.954719 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.954702    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-podinfo\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.962947 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.962923    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-token\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:47.963326 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:47.963306    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl9zf\" (UniqueName: \"kubernetes.io/projected/7bc05684-948b-414b-9a71-ba8f47eaba93-kube-api-access-hl9zf\") pod \"e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") " pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:48.151689 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:48.151594    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:48.426437 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:48.426391    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"]
Sep 11 22:01:48.428619 ip-10-0-142-211 kubenswrapper[2544]: W0911 22:01:48.428595    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc05684_948b_414b_9a71_ba8f47eaba93.slice/crio-f2b57ff7327bf1e2d91123956f525ccbf84562cff2a4e14201ed169231b676dd WatchSource:0}: Error finding container f2b57ff7327bf1e2d91123956f525ccbf84562cff2a4e14201ed169231b676dd: Status 404 returned error can't find the container with id f2b57ff7327bf1e2d91123956f525ccbf84562cff2a4e14201ed169231b676dd
Sep 11 22:01:48.430841 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:48.430809    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 22:01:48.430908 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:48.430894    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 22:01:48.430945 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:48.430921    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 22:01:48.704082 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:48.703993    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq" event={"ID":"7bc05684-948b-414b-9a71-ba8f47eaba93","Type":"ContainerStarted","Data":"e10a3a133e13ffe61ec6ad3559f329a52c816bcebedda7acb7357627e30c7af9"}
Sep 11 22:01:48.704082 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:48.704047    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq" event={"ID":"7bc05684-948b-414b-9a71-ba8f47eaba93","Type":"ContainerStarted","Data":"f2b57ff7327bf1e2d91123956f525ccbf84562cff2a4e14201ed169231b676dd"}
Sep 11 22:01:48.724967 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:48.724903    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq" podStartSLOduration=1.724887367 podStartE2EDuration="1.724887367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:01:48.723299084 +0000 UTC m=+1871.692715905" watchObservedRunningTime="2026-09-11 22:01:48.724887367 +0000 UTC m=+1871.694304187"
Sep 11 22:01:49.151997 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:49.151958    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:49.157204 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:49.157179    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:49.708461 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:49.708430    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:49.709379 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:49.709361    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:01:53.685791 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.685753    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6"]
Sep 11 22:01:53.689070 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.689044    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6"
Sep 11 22:01:53.691407 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.691384    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-077193\""
Sep 11 22:01:53.699720 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.699691    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6"]
Sep 11 22:01:53.719898 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.719858    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp"]
Sep 11 22:01:53.722959 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.722936    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp"
Sep 11 22:01:53.725334 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.725298    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-077193-dockercfg-g7bx8\""
Sep 11 22:01:53.734385 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.734361    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp"]
Sep 11 22:01:53.805292 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.805249    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b72483-ec76-4bc2-a8ea-fdc16691d30e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6\" (UID: \"85b72483-ec76-4bc2-a8ea-fdc16691d30e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6"
Sep 11 22:01:53.805532 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.805301    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txb5v\" (UniqueName: \"kubernetes.io/projected/85b72483-ec76-4bc2-a8ea-fdc16691d30e-kube-api-access-txb5v\") pod \"payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6\" (UID: \"85b72483-ec76-4bc2-a8ea-fdc16691d30e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6"
Sep 11 22:01:53.805532 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.805337    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703b93ef-d00e-4e9e-9e2e-6c407c7539d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp\" (UID: \"703b93ef-d00e-4e9e-9e2e-6c407c7539d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp"
Sep 11 22:01:53.805532 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.805387    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7bfp\" (UniqueName: \"kubernetes.io/projected/703b93ef-d00e-4e9e-9e2e-6c407c7539d8-kube-api-access-w7bfp\") pod \"payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp\" (UID: \"703b93ef-d00e-4e9e-9e2e-6c407c7539d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp"
Sep 11 22:01:53.906763 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.906721    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b72483-ec76-4bc2-a8ea-fdc16691d30e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6\" (UID: \"85b72483-ec76-4bc2-a8ea-fdc16691d30e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6"
Sep 11 22:01:53.906763 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.906760    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txb5v\" (UniqueName: \"kubernetes.io/projected/85b72483-ec76-4bc2-a8ea-fdc16691d30e-kube-api-access-txb5v\") pod \"payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6\" (UID: \"85b72483-ec76-4bc2-a8ea-fdc16691d30e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6"
Sep 11 22:01:53.907045 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.906777    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703b93ef-d00e-4e9e-9e2e-6c407c7539d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp\" (UID: \"703b93ef-d00e-4e9e-9e2e-6c407c7539d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp"
Sep 11 22:01:53.907045 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.906808    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7bfp\" (UniqueName: \"kubernetes.io/projected/703b93ef-d00e-4e9e-9e2e-6c407c7539d8-kube-api-access-w7bfp\") pod \"payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp\" (UID: \"703b93ef-d00e-4e9e-9e2e-6c407c7539d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp"
Sep 11 22:01:53.907576 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.907552    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b72483-ec76-4bc2-a8ea-fdc16691d30e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6\" (UID: \"85b72483-ec76-4bc2-a8ea-fdc16691d30e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6"
Sep 11 22:01:53.907672 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.907595    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703b93ef-d00e-4e9e-9e2e-6c407c7539d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp\" (UID: \"703b93ef-d00e-4e9e-9e2e-6c407c7539d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp"
Sep 11 22:01:53.923612 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.923571    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7bfp\" (UniqueName: \"kubernetes.io/projected/703b93ef-d00e-4e9e-9e2e-6c407c7539d8-kube-api-access-w7bfp\") pod \"payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp\" (UID: \"703b93ef-d00e-4e9e-9e2e-6c407c7539d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp"
Sep 11 22:01:53.936980 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.936916    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txb5v\" (UniqueName: \"kubernetes.io/projected/85b72483-ec76-4bc2-a8ea-fdc16691d30e-kube-api-access-txb5v\") pod \"payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6\" (UID: \"85b72483-ec76-4bc2-a8ea-fdc16691d30e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6"
Sep 11 22:01:53.999448 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:53.999406    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6"
Sep 11 22:01:54.033399 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:54.033361    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp"
Sep 11 22:01:54.271565 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:54.271536    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6"]
Sep 11 22:01:54.273212 ip-10-0-142-211 kubenswrapper[2544]: W0911 22:01:54.273191    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b72483_ec76_4bc2_a8ea_fdc16691d30e.slice/crio-26f79224dc71536a85d3e9f01cd20f83455ca95491f8071f1ba6b1d253ab0a41 WatchSource:0}: Error finding container 26f79224dc71536a85d3e9f01cd20f83455ca95491f8071f1ba6b1d253ab0a41: Status 404 returned error can't find the container with id 26f79224dc71536a85d3e9f01cd20f83455ca95491f8071f1ba6b1d253ab0a41
Sep 11 22:01:54.314807 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:54.314782    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp"]
Sep 11 22:01:54.317021 ip-10-0-142-211 kubenswrapper[2544]: W0911 22:01:54.316996    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703b93ef_d00e_4e9e_9e2e_6c407c7539d8.slice/crio-e75341598b945ff649d9d5a8f1cec58f3994c75f2b8ac0e2ac3b7158b24bdfc6 WatchSource:0}: Error finding container e75341598b945ff649d9d5a8f1cec58f3994c75f2b8ac0e2ac3b7158b24bdfc6: Status 404 returned error can't find the container with id e75341598b945ff649d9d5a8f1cec58f3994c75f2b8ac0e2ac3b7158b24bdfc6
Sep 11 22:01:54.732554 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:54.732515    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6" event={"ID":"85b72483-ec76-4bc2-a8ea-fdc16691d30e","Type":"ContainerStarted","Data":"8c0d1cc2dfd765c347d178c6606741947ea46bcecd70476bc1d3d26450557e50"}
Sep 11 22:01:54.733024 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:54.732581    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6" event={"ID":"85b72483-ec76-4bc2-a8ea-fdc16691d30e","Type":"ContainerStarted","Data":"26f79224dc71536a85d3e9f01cd20f83455ca95491f8071f1ba6b1d253ab0a41"}
Sep 11 22:01:54.733024 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:54.732724    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6"
Sep 11 22:01:54.734431 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:54.734402    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp" event={"ID":"703b93ef-d00e-4e9e-9e2e-6c407c7539d8","Type":"ContainerStarted","Data":"9b6b01a9f6dd6e705ecf101f8830fcb88f9c0107bb4c3e25f26fcda64b3a5274"}
Sep 11 22:01:54.734569 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:54.734451    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp"
Sep 11 22:01:54.734569 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:54.734472    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp" event={"ID":"703b93ef-d00e-4e9e-9e2e-6c407c7539d8","Type":"ContainerStarted","Data":"e75341598b945ff649d9d5a8f1cec58f3994c75f2b8ac0e2ac3b7158b24bdfc6"}
Sep 11 22:01:54.749510 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:54.749437    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6" podStartSLOduration=1.749418088 podStartE2EDuration="1.749418088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:01:54.746919324 +0000 UTC m=+1877.716336157" watchObservedRunningTime="2026-09-11 22:01:54.749418088 +0000 UTC m=+1877.718834909"
Sep 11 22:01:54.762113 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:01:54.762054    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp" podStartSLOduration=1.762035153 podStartE2EDuration="1.762035153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:01:54.761625293 +0000 UTC m=+1877.731042117" watchObservedRunningTime="2026-09-11 22:01:54.762035153 +0000 UTC m=+1877.731451974"
Sep 11 22:02:05.741866 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:05.741833    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp"
Sep 11 22:02:05.742342 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:05.741909    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6"
Sep 11 22:02:08.807018 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:08.806934    2544 generic.go:356] "Generic (PLEG): container finished" podID="7bc05684-948b-414b-9a71-ba8f47eaba93" containerID="e10a3a133e13ffe61ec6ad3559f329a52c816bcebedda7acb7357627e30c7af9" exitCode=0
Sep 11 22:02:08.807018 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:08.806991    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq" event={"ID":"7bc05684-948b-414b-9a71-ba8f47eaba93","Type":"ContainerDied","Data":"e10a3a133e13ffe61ec6ad3559f329a52c816bcebedda7acb7357627e30c7af9"}
Sep 11 22:02:08.807470 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:08.807423    2544 scope.go:117] "RemoveContainer" containerID="e10a3a133e13ffe61ec6ad3559f329a52c816bcebedda7acb7357627e30c7af9"
Sep 11 22:02:08.808498 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:08.808470    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 22:02:08.808556 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:08.808539    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 22:02:08.808594 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:08.808568    2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 22:02:09.814672 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:09.814635    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq" event={"ID":"7bc05684-948b-414b-9a71-ba8f47eaba93","Type":"ContainerStarted","Data":"79445d3b6e1e618a4b6312fccbbe4874c5053310380005282d50f180ed097104"}
Sep 11 22:02:10.152200 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:10.152088    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:02:10.152407 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:10.152349    2544 patch_prober.go:28] interesting pod/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 11 22:02:10.152482 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:10.152407    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq" podUID="7bc05684-948b-414b-9a71-ba8f47eaba93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 11 22:02:11.151946 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:11.151908    2544 patch_prober.go:28] interesting pod/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 11 22:02:11.152335 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:11.151969    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq" podUID="7bc05684-948b-414b-9a71-ba8f47eaba93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 11 22:02:12.152260 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:12.152198    2544 patch_prober.go:28] interesting pod/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 11 22:02:12.152653 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:12.152283    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq" podUID="7bc05684-948b-414b-9a71-ba8f47eaba93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 11 22:02:13.152198 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:13.152154    2544 patch_prober.go:28] interesting pod/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 11 22:02:13.152400 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:13.152248    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq" podUID="7bc05684-948b-414b-9a71-ba8f47eaba93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 11 22:02:14.152627 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:14.152584    2544 patch_prober.go:28] interesting pod/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 11 22:02:14.153019 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:14.152657    2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq" podUID="7bc05684-948b-414b-9a71-ba8f47eaba93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 11 22:02:15.189903 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:15.189828    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:02:15.190303 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:15.190069    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:02:15.190867 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:15.190849    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:02:20.830935 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:20.830906    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp_703b93ef-d00e-4e9e-9e2e-6c407c7539d8/payload-processing/0.log"
Sep 11 22:02:24.955955 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:24.955915    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"]
Sep 11 22:02:24.958490 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:24.958468    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:24.961155 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:24.961134    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6a5690\"/\"openshift-service-ca.crt\""
Sep 11 22:02:24.962279 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:24.962192    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6a5690\"/\"default-dockercfg-vgtfr\""
Sep 11 22:02:24.962279 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:24.962212    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6a5690\"/\"kube-root-ca.crt\""
Sep 11 22:02:24.962279 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:24.962212    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6a5690\"/\"ipp-route-6a5690-kserve-self-signed-certs\""
Sep 11 22:02:24.969684 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:24.969662    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"]
Sep 11 22:02:25.092154 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.092120    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-tls-certs\") pod \"ipp-route-6a5690-kserve-649cb867f9-wvr5s\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") " pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.092357 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.092175    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46t4d\" (UniqueName: \"kubernetes.io/projected/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-kube-api-access-46t4d\") pod \"ipp-route-6a5690-kserve-649cb867f9-wvr5s\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") " pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.092357 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.092251    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-dshm\") pod \"ipp-route-6a5690-kserve-649cb867f9-wvr5s\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") " pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.092357 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.092298    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-home\") pod \"ipp-route-6a5690-kserve-649cb867f9-wvr5s\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") " pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.092357 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.092345    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-model-cache\") pod \"ipp-route-6a5690-kserve-649cb867f9-wvr5s\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") " pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.092571 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.092403    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-tmp-dir\") pod \"ipp-route-6a5690-kserve-649cb867f9-wvr5s\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") " pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.193120 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.193087    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-tmp-dir\") pod \"ipp-route-6a5690-kserve-649cb867f9-wvr5s\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") " pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.193351 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.193140    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-tls-certs\") pod \"ipp-route-6a5690-kserve-649cb867f9-wvr5s\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") " pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.193351 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.193172    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46t4d\" (UniqueName: \"kubernetes.io/projected/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-kube-api-access-46t4d\") pod \"ipp-route-6a5690-kserve-649cb867f9-wvr5s\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") " pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.193351 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.193206    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-dshm\") pod \"ipp-route-6a5690-kserve-649cb867f9-wvr5s\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") " pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.193351 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.193257    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-home\") pod \"ipp-route-6a5690-kserve-649cb867f9-wvr5s\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") " pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.193351 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.193308    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-model-cache\") pod \"ipp-route-6a5690-kserve-649cb867f9-wvr5s\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") " pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.193616 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.193559    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-tmp-dir\") pod \"ipp-route-6a5690-kserve-649cb867f9-wvr5s\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") " pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.193683 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.193618    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-model-cache\") pod \"ipp-route-6a5690-kserve-649cb867f9-wvr5s\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") " pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.193683 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.193648    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-home\") pod \"ipp-route-6a5690-kserve-649cb867f9-wvr5s\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") " pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.195360 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.195337    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-dshm\") pod \"ipp-route-6a5690-kserve-649cb867f9-wvr5s\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") " pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.195617 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.195600    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-tls-certs\") pod \"ipp-route-6a5690-kserve-649cb867f9-wvr5s\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") " pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.201310 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.201282    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46t4d\" (UniqueName: \"kubernetes.io/projected/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-kube-api-access-46t4d\") pod \"ipp-route-6a5690-kserve-649cb867f9-wvr5s\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") " pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.268735 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.268705    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:25.535887 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.535862    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"]
Sep 11 22:02:25.537317 ip-10-0-142-211 kubenswrapper[2544]: W0911 22:02:25.537282    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3abe6c2b_e422_4af6_b419_ae5e138e3cb7.slice/crio-681e55a6862763c30435ac96ce626179e22fb239ae6d9c6ec81d4b7b12bb3387 WatchSource:0}: Error finding container 681e55a6862763c30435ac96ce626179e22fb239ae6d9c6ec81d4b7b12bb3387: Status 404 returned error can't find the container with id 681e55a6862763c30435ac96ce626179e22fb239ae6d9c6ec81d4b7b12bb3387
Sep 11 22:02:25.899296 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.899188    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s" event={"ID":"3abe6c2b-e422-4af6-b419-ae5e138e3cb7","Type":"ContainerStarted","Data":"9b2155ff62ef0f59de2e41018e5bb19ac75d5c075d58b0f012d9432bd70605ea"}
Sep 11 22:02:25.899296 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.899248    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s" event={"ID":"3abe6c2b-e422-4af6-b419-ae5e138e3cb7","Type":"ContainerStarted","Data":"681e55a6862763c30435ac96ce626179e22fb239ae6d9c6ec81d4b7b12bb3387"}
Sep 11 22:02:25.918293 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:25.918242    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s" podStartSLOduration=1.918214036 podStartE2EDuration="1.918214036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:02:25.916321124 +0000 UTC m=+1908.885737944" watchObservedRunningTime="2026-09-11 22:02:25.918214036 +0000 UTC m=+1908.887630856"
Sep 11 22:02:35.269117 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:35.269069    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:35.269555 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:35.269138    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:35.281670 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:35.281645    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:35.959231 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:35.959180    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:02:37.970522 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:37.970491    2544 scope.go:117] "RemoveContainer" containerID="7aaab3341bf608d2932dace01aeac27457a2f4162435e8cfccb6302dcdfe4172"
Sep 11 22:02:37.978739 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:37.978719    2544 scope.go:117] "RemoveContainer" containerID="68158084d6268feedd5de2e4e530b220a02d1fb3d126064271fda92bdedaf8b4"
Sep 11 22:02:37.986180 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:37.986149    2544 scope.go:117] "RemoveContainer" containerID="92c1caf7863486efdbd921eac72e3a454d394584f8def156dab28ceb405c5735"
Sep 11 22:02:56.900850 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:02:56.900818    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp_703b93ef-d00e-4e9e-9e2e-6c407c7539d8/payload-processing/0.log"
Sep 11 22:03:26.046368 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:26.046329    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk"]
Sep 11 22:03:26.048785 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:26.048769    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk"
Sep 11 22:03:26.051192 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:26.051165    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:03:26.051351 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:26.051192    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 22:03:26.057736 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:26.057702    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk"]
Sep 11 22:03:26.215893 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:26.215851    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b95f4ba-aaa5-47b1-9765-f62075d898d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk\" (UID: \"0b95f4ba-aaa5-47b1-9765-f62075d898d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk"
Sep 11 22:03:26.316586 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:26.316498    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b95f4ba-aaa5-47b1-9765-f62075d898d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk\" (UID: \"0b95f4ba-aaa5-47b1-9765-f62075d898d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk"
Sep 11 22:03:26.317128 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:26.317109    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b95f4ba-aaa5-47b1-9765-f62075d898d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk\" (UID: \"0b95f4ba-aaa5-47b1-9765-f62075d898d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk"
Sep 11 22:03:26.360610 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:26.360571    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk"
Sep 11 22:03:26.627621 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:26.627594    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk"]
Sep 11 22:03:26.629683 ip-10-0-142-211 kubenswrapper[2544]: W0911 22:03:26.629655    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b95f4ba_aaa5_47b1_9765_f62075d898d0.slice/crio-dfbcc2658a73f006d08fbcca3744164b29ada782a67ed1e57ed868e96533b9d0 WatchSource:0}: Error finding container dfbcc2658a73f006d08fbcca3744164b29ada782a67ed1e57ed868e96533b9d0: Status 404 returned error can't find the container with id dfbcc2658a73f006d08fbcca3744164b29ada782a67ed1e57ed868e96533b9d0
Sep 11 22:03:27.180989 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:27.180954    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk" event={"ID":"0b95f4ba-aaa5-47b1-9765-f62075d898d0","Type":"ContainerStarted","Data":"ed04e48b3f0681d36f24f2b6125e5573a0e032d995382b91c0cc52dfbad03d4b"}
Sep 11 22:03:27.181435 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:27.181014    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk" event={"ID":"0b95f4ba-aaa5-47b1-9765-f62075d898d0","Type":"ContainerStarted","Data":"dfbcc2658a73f006d08fbcca3744164b29ada782a67ed1e57ed868e96533b9d0"}
Sep 11 22:03:27.198270 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:27.198199    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk" podStartSLOduration=1.198184602 podStartE2EDuration="1.198184602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:03:27.197743876 +0000 UTC m=+1970.167160695" watchObservedRunningTime="2026-09-11 22:03:27.198184602 +0000 UTC m=+1970.167601422"
Sep 11 22:03:28.187953 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:28.187917    2544 generic.go:356] "Generic (PLEG): container finished" podID="0b95f4ba-aaa5-47b1-9765-f62075d898d0" containerID="ed04e48b3f0681d36f24f2b6125e5573a0e032d995382b91c0cc52dfbad03d4b" exitCode=0
Sep 11 22:03:28.188368 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:28.187978    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk" event={"ID":"0b95f4ba-aaa5-47b1-9765-f62075d898d0","Type":"ContainerDied","Data":"ed04e48b3f0681d36f24f2b6125e5573a0e032d995382b91c0cc52dfbad03d4b"}
Sep 11 22:03:29.323405 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:29.323379    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk"
Sep 11 22:03:29.445856 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:29.445761    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b95f4ba-aaa5-47b1-9765-f62075d898d0-maas-api-service-ca\") pod \"0b95f4ba-aaa5-47b1-9765-f62075d898d0\" (UID: \"0b95f4ba-aaa5-47b1-9765-f62075d898d0\") "
Sep 11 22:03:29.446173 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:29.446152    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b95f4ba-aaa5-47b1-9765-f62075d898d0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b95f4ba-aaa5-47b1-9765-f62075d898d0" (UID: "0b95f4ba-aaa5-47b1-9765-f62075d898d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:03:29.546995 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:29.546924    2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b95f4ba-aaa5-47b1-9765-f62075d898d0-maas-api-service-ca\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:03:30.198610 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:30.198571    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk" event={"ID":"0b95f4ba-aaa5-47b1-9765-f62075d898d0","Type":"ContainerDied","Data":"dfbcc2658a73f006d08fbcca3744164b29ada782a67ed1e57ed868e96533b9d0"}
Sep 11 22:03:30.198782 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:30.198627    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfbcc2658a73f006d08fbcca3744164b29ada782a67ed1e57ed868e96533b9d0"
Sep 11 22:03:30.198782 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:30.198591    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk"
Sep 11 22:03:32.368237 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:32.368183    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk"]
Sep 11 22:03:32.370510 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:32.370487    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3b9cac-94jdk"]
Sep 11 22:03:33.591631 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:33.591599    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b95f4ba-aaa5-47b1-9765-f62075d898d0" path="/var/lib/kubelet/pods/0b95f4ba-aaa5-47b1-9765-f62075d898d0/volumes"
Sep 11 22:03:38.013259 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:38.012256    2544 scope.go:117] "RemoveContainer" containerID="211f8c4809bdc39ac2e5744e79e47d84de580dc590ffeb73f86e54e420660683"
Sep 11 22:03:38.022945 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:38.022922    2544 scope.go:117] "RemoveContainer" containerID="3b32b92f9adab9a46b970f1ea2996bdf042879a71c91bfc3ade0d0110629316a"
Sep 11 22:03:42.662362 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:42.662329    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a5690-c64ng"]
Sep 11 22:03:42.662732 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:42.662700    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b95f4ba-aaa5-47b1-9765-f62075d898d0" containerName="revoke-keys"
Sep 11 22:03:42.662732 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:42.662712    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b95f4ba-aaa5-47b1-9765-f62075d898d0" containerName="revoke-keys"
Sep 11 22:03:42.662811 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:42.662771    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b95f4ba-aaa5-47b1-9765-f62075d898d0" containerName="revoke-keys"
Sep 11 22:03:42.664585 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:42.664569    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a5690-c64ng"
Sep 11 22:03:42.666855 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:42.666830    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:03:42.666944 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:42.666830    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 22:03:42.674278 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:42.674249    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a5690-c64ng"]
Sep 11 22:03:42.764758 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:42.764715    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5519a641-a5b8-4dc3-ad6a-e16b55176682-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6a5690-c64ng\" (UID: \"5519a641-a5b8-4dc3-ad6a-e16b55176682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a5690-c64ng"
Sep 11 22:03:42.865880 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:42.865833    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5519a641-a5b8-4dc3-ad6a-e16b55176682-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6a5690-c64ng\" (UID: \"5519a641-a5b8-4dc3-ad6a-e16b55176682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a5690-c64ng"
Sep 11 22:03:42.866565 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:42.866543    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5519a641-a5b8-4dc3-ad6a-e16b55176682-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6a5690-c64ng\" (UID: \"5519a641-a5b8-4dc3-ad6a-e16b55176682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a5690-c64ng"
Sep 11 22:03:42.975009 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:42.974978    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a5690-c64ng"
Sep 11 22:03:43.240946 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:43.240917    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a5690-c64ng"]
Sep 11 22:03:43.242892 ip-10-0-142-211 kubenswrapper[2544]: W0911 22:03:43.242864    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5519a641_a5b8_4dc3_ad6a_e16b55176682.slice/crio-2fb5059d5ca6059bc8c38d37ca81088a782fa0d52d9fbfdb9bc9cff341d6e702 WatchSource:0}: Error finding container 2fb5059d5ca6059bc8c38d37ca81088a782fa0d52d9fbfdb9bc9cff341d6e702: Status 404 returned error can't find the container with id 2fb5059d5ca6059bc8c38d37ca81088a782fa0d52d9fbfdb9bc9cff341d6e702
Sep 11 22:03:43.266646 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:43.266615    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a5690-c64ng" event={"ID":"5519a641-a5b8-4dc3-ad6a-e16b55176682","Type":"ContainerStarted","Data":"2fb5059d5ca6059bc8c38d37ca81088a782fa0d52d9fbfdb9bc9cff341d6e702"}
Sep 11 22:03:44.272809 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:44.272769    2544 generic.go:356] "Generic (PLEG): container finished" podID="5519a641-a5b8-4dc3-ad6a-e16b55176682" containerID="81a76fe1d0e418c94502d70aa3ac5b8d4bdf102514175db551175ae047325aa5" exitCode=0
Sep 11 22:03:44.273214 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:44.272819    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a5690-c64ng" event={"ID":"5519a641-a5b8-4dc3-ad6a-e16b55176682","Type":"ContainerDied","Data":"81a76fe1d0e418c94502d70aa3ac5b8d4bdf102514175db551175ae047325aa5"}
Sep 11 22:03:45.410129 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:45.410102    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a5690-c64ng"
Sep 11 22:03:45.492215 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:45.492173    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5519a641-a5b8-4dc3-ad6a-e16b55176682-maas-api-service-ca\") pod \"5519a641-a5b8-4dc3-ad6a-e16b55176682\" (UID: \"5519a641-a5b8-4dc3-ad6a-e16b55176682\") "
Sep 11 22:03:45.492563 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:45.492538    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5519a641-a5b8-4dc3-ad6a-e16b55176682-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5519a641-a5b8-4dc3-ad6a-e16b55176682" (UID: "5519a641-a5b8-4dc3-ad6a-e16b55176682"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:03:45.593629 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:45.593540    2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5519a641-a5b8-4dc3-ad6a-e16b55176682-maas-api-service-ca\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:03:46.283551 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:46.283523    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a5690-c64ng"
Sep 11 22:03:46.283839 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:46.283569    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a5690-c64ng" event={"ID":"5519a641-a5b8-4dc3-ad6a-e16b55176682","Type":"ContainerDied","Data":"2fb5059d5ca6059bc8c38d37ca81088a782fa0d52d9fbfdb9bc9cff341d6e702"}
Sep 11 22:03:46.283839 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:46.283624    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fb5059d5ca6059bc8c38d37ca81088a782fa0d52d9fbfdb9bc9cff341d6e702"
Sep 11 22:03:58.378603 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:58.378569    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a5690-c64ng"]
Sep 11 22:03:58.380248 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:58.380214    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a5690-c64ng"]
Sep 11 22:03:59.592056 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:03:59.592022    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5519a641-a5b8-4dc3-ad6a-e16b55176682" path="/var/lib/kubelet/pods/5519a641-a5b8-4dc3-ad6a-e16b55176682/volumes"
Sep 11 22:04:03.720246 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:03.720200    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"]
Sep 11 22:04:18.739358 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:18.739320    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s" podUID="3abe6c2b-e422-4af6-b419-ae5e138e3cb7" containerName="main" containerID="cri-o://9b2155ff62ef0f59de2e41018e5bb19ac75d5c075d58b0f012d9432bd70605ea" gracePeriod=45
Sep 11 22:04:18.988737 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:18.988709    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:04:19.091349 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.091260    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-model-cache\") pod \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") "
Sep 11 22:04:19.091513 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.091365    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-home\") pod \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") "
Sep 11 22:04:19.091513 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.091401    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-dshm\") pod \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") "
Sep 11 22:04:19.091513 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.091429    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-tls-certs\") pod \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") "
Sep 11 22:04:19.091513 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.091463    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46t4d\" (UniqueName: \"kubernetes.io/projected/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-kube-api-access-46t4d\") pod \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") "
Sep 11 22:04:19.091513 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.091501    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-tmp-dir\") pod \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\" (UID: \"3abe6c2b-e422-4af6-b419-ae5e138e3cb7\") "
Sep 11 22:04:19.091762 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.091575    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-model-cache" (OuterVolumeSpecName: "model-cache") pod "3abe6c2b-e422-4af6-b419-ae5e138e3cb7" (UID: "3abe6c2b-e422-4af6-b419-ae5e138e3cb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:04:19.091762 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.091672    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-home" (OuterVolumeSpecName: "home") pod "3abe6c2b-e422-4af6-b419-ae5e138e3cb7" (UID: "3abe6c2b-e422-4af6-b419-ae5e138e3cb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:04:19.091873 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.091816    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:04:19.091873 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.091829    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:04:19.091960 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.091866    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3abe6c2b-e422-4af6-b419-ae5e138e3cb7" (UID: "3abe6c2b-e422-4af6-b419-ae5e138e3cb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:04:19.093677 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.093647    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3abe6c2b-e422-4af6-b419-ae5e138e3cb7" (UID: "3abe6c2b-e422-4af6-b419-ae5e138e3cb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:04:19.093677 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.093660    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-dshm" (OuterVolumeSpecName: "dshm") pod "3abe6c2b-e422-4af6-b419-ae5e138e3cb7" (UID: "3abe6c2b-e422-4af6-b419-ae5e138e3cb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:04:19.093823 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.093685    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-kube-api-access-46t4d" (OuterVolumeSpecName: "kube-api-access-46t4d") pod "3abe6c2b-e422-4af6-b419-ae5e138e3cb7" (UID: "3abe6c2b-e422-4af6-b419-ae5e138e3cb7"). InnerVolumeSpecName "kube-api-access-46t4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:04:19.193074 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.193037    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:04:19.193074 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.193067    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:04:19.193074 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.193081    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46t4d\" (UniqueName: \"kubernetes.io/projected/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-kube-api-access-46t4d\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:04:19.193369 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.193093    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3abe6c2b-e422-4af6-b419-ae5e138e3cb7-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:04:19.445189 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.445101    2544 generic.go:356] "Generic (PLEG): container finished" podID="3abe6c2b-e422-4af6-b419-ae5e138e3cb7" containerID="9b2155ff62ef0f59de2e41018e5bb19ac75d5c075d58b0f012d9432bd70605ea" exitCode=0
Sep 11 22:04:19.445189 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.445145    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"
Sep 11 22:04:19.445412 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.445182    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s" event={"ID":"3abe6c2b-e422-4af6-b419-ae5e138e3cb7","Type":"ContainerDied","Data":"9b2155ff62ef0f59de2e41018e5bb19ac75d5c075d58b0f012d9432bd70605ea"}
Sep 11 22:04:19.445412 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.445234    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s" event={"ID":"3abe6c2b-e422-4af6-b419-ae5e138e3cb7","Type":"ContainerDied","Data":"681e55a6862763c30435ac96ce626179e22fb239ae6d9c6ec81d4b7b12bb3387"}
Sep 11 22:04:19.445412 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.445268    2544 scope.go:117] "RemoveContainer" containerID="9b2155ff62ef0f59de2e41018e5bb19ac75d5c075d58b0f012d9432bd70605ea"
Sep 11 22:04:19.453780 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.453759    2544 scope.go:117] "RemoveContainer" containerID="9b2155ff62ef0f59de2e41018e5bb19ac75d5c075d58b0f012d9432bd70605ea"
Sep 11 22:04:19.454030 ip-10-0-142-211 kubenswrapper[2544]: E0911 22:04:19.454013    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2155ff62ef0f59de2e41018e5bb19ac75d5c075d58b0f012d9432bd70605ea\": container with ID starting with 9b2155ff62ef0f59de2e41018e5bb19ac75d5c075d58b0f012d9432bd70605ea not found: ID does not exist" containerID="9b2155ff62ef0f59de2e41018e5bb19ac75d5c075d58b0f012d9432bd70605ea"
Sep 11 22:04:19.454083 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.454038    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2155ff62ef0f59de2e41018e5bb19ac75d5c075d58b0f012d9432bd70605ea"} err="failed to get container status \"9b2155ff62ef0f59de2e41018e5bb19ac75d5c075d58b0f012d9432bd70605ea\": rpc error: code = NotFound desc = could not find container \"9b2155ff62ef0f59de2e41018e5bb19ac75d5c075d58b0f012d9432bd70605ea\": container with ID starting with 9b2155ff62ef0f59de2e41018e5bb19ac75d5c075d58b0f012d9432bd70605ea not found: ID does not exist"
Sep 11 22:04:19.466306 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.466277    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"]
Sep 11 22:04:19.471655 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.471632    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-6a5690/ipp-route-6a5690-kserve-649cb867f9-wvr5s"]
Sep 11 22:04:19.592496 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:19.592461    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3abe6c2b-e422-4af6-b419-ae5e138e3cb7" path="/var/lib/kubelet/pods/3abe6c2b-e422-4af6-b419-ae5e138e3cb7/volumes"
Sep 11 22:04:47.475991 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:47.475953    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh"]
Sep 11 22:04:47.476422 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:47.476326    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5519a641-a5b8-4dc3-ad6a-e16b55176682" containerName="revoke-keys"
Sep 11 22:04:47.476422 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:47.476339    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="5519a641-a5b8-4dc3-ad6a-e16b55176682" containerName="revoke-keys"
Sep 11 22:04:47.476422 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:47.476349    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3abe6c2b-e422-4af6-b419-ae5e138e3cb7" containerName="main"
Sep 11 22:04:47.476422 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:47.476355    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="3abe6c2b-e422-4af6-b419-ae5e138e3cb7" containerName="main"
Sep 11 22:04:47.476422 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:47.476412    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="3abe6c2b-e422-4af6-b419-ae5e138e3cb7" containerName="main"
Sep 11 22:04:47.476422 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:47.476423    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="5519a641-a5b8-4dc3-ad6a-e16b55176682" containerName="revoke-keys"
Sep 11 22:04:47.479448 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:47.479431    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh"
Sep 11 22:04:47.481764 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:47.481743    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 22:04:47.481890 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:47.481781    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:04:47.486795 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:47.486772    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh"]
Sep 11 22:04:47.543286 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:47.543242    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31206583-b73b-4dee-ba30-a2a822ca3733-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh\" (UID: \"31206583-b73b-4dee-ba30-a2a822ca3733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh"
Sep 11 22:04:47.644703 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:47.644661    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31206583-b73b-4dee-ba30-a2a822ca3733-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh\" (UID: \"31206583-b73b-4dee-ba30-a2a822ca3733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh"
Sep 11 22:04:47.645370 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:47.645346    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31206583-b73b-4dee-ba30-a2a822ca3733-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh\" (UID: \"31206583-b73b-4dee-ba30-a2a822ca3733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh"
Sep 11 22:04:47.791738 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:47.791654    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh"
Sep 11 22:04:48.061758 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:48.061735    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh"]
Sep 11 22:04:48.063897 ip-10-0-142-211 kubenswrapper[2544]: W0911 22:04:48.063872    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31206583_b73b_4dee_ba30_a2a822ca3733.slice/crio-e44a566130fdf2643474115888d672e0af6078875eb333df3b909dcc80d83a05 WatchSource:0}: Error finding container e44a566130fdf2643474115888d672e0af6078875eb333df3b909dcc80d83a05: Status 404 returned error can't find the container with id e44a566130fdf2643474115888d672e0af6078875eb333df3b909dcc80d83a05
Sep 11 22:04:48.584407 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:48.584313    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh" event={"ID":"31206583-b73b-4dee-ba30-a2a822ca3733","Type":"ContainerStarted","Data":"adcc6db074953194ea7bf11d45d880518df28f4f6fe59d906f77b1fe1d9562e2"}
Sep 11 22:04:48.584407 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:48.584368    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh" event={"ID":"31206583-b73b-4dee-ba30-a2a822ca3733","Type":"ContainerStarted","Data":"e44a566130fdf2643474115888d672e0af6078875eb333df3b909dcc80d83a05"}
Sep 11 22:04:48.600750 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:48.600701    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh" podStartSLOduration=1.600688145 podStartE2EDuration="1.600688145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:48.598654975 +0000 UTC m=+2051.568071794" watchObservedRunningTime="2026-09-11 22:04:48.600688145 +0000 UTC m=+2051.570104986"
Sep 11 22:04:49.590734 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:49.590693    2544 generic.go:356] "Generic (PLEG): container finished" podID="31206583-b73b-4dee-ba30-a2a822ca3733" containerID="adcc6db074953194ea7bf11d45d880518df28f4f6fe59d906f77b1fe1d9562e2" exitCode=0
Sep 11 22:04:49.591744 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:49.591718    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh" event={"ID":"31206583-b73b-4dee-ba30-a2a822ca3733","Type":"ContainerDied","Data":"adcc6db074953194ea7bf11d45d880518df28f4f6fe59d906f77b1fe1d9562e2"}
Sep 11 22:04:50.743981 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:50.743959    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh"
Sep 11 22:04:50.875322 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:50.875204    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31206583-b73b-4dee-ba30-a2a822ca3733-maas-api-service-ca\") pod \"31206583-b73b-4dee-ba30-a2a822ca3733\" (UID: \"31206583-b73b-4dee-ba30-a2a822ca3733\") "
Sep 11 22:04:50.875578 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:50.875556    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31206583-b73b-4dee-ba30-a2a822ca3733-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "31206583-b73b-4dee-ba30-a2a822ca3733" (UID: "31206583-b73b-4dee-ba30-a2a822ca3733"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:04:50.976440 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:50.976406    2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31206583-b73b-4dee-ba30-a2a822ca3733-maas-api-service-ca\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:04:51.602199 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:51.602159    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh" event={"ID":"31206583-b73b-4dee-ba30-a2a822ca3733","Type":"ContainerDied","Data":"e44a566130fdf2643474115888d672e0af6078875eb333df3b909dcc80d83a05"}
Sep 11 22:04:51.602199 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:51.602180    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh"
Sep 11 22:04:51.602374 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:51.602231    2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e44a566130fdf2643474115888d672e0af6078875eb333df3b909dcc80d83a05"
Sep 11 22:04:57.938421 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:57.938386    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp"]
Sep 11 22:04:57.938912 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:57.938607    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp" podUID="703b93ef-d00e-4e9e-9e2e-6c407c7539d8" containerName="payload-processing" containerID="cri-o://9b6b01a9f6dd6e705ecf101f8830fcb88f9c0107bb4c3e25f26fcda64b3a5274" gracePeriod=30
Sep 11 22:04:57.945820 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:57.945732    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6"]
Sep 11 22:04:57.945977 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:57.945949    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6" podUID="85b72483-ec76-4bc2-a8ea-fdc16691d30e" containerName="payload-pre-processing" containerID="cri-o://8c0d1cc2dfd765c347d178c6606741947ea46bcecd70476bc1d3d26450557e50" gracePeriod=30
Sep 11 22:04:58.091588 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.091553    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh"]
Sep 11 22:04:58.094249 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.094205    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-077193-4pbwh"]
Sep 11 22:04:58.223942 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.223912    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp"
Sep 11 22:04:58.226543 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.226510    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6"
Sep 11 22:04:58.347449 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.347411    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txb5v\" (UniqueName: \"kubernetes.io/projected/85b72483-ec76-4bc2-a8ea-fdc16691d30e-kube-api-access-txb5v\") pod \"85b72483-ec76-4bc2-a8ea-fdc16691d30e\" (UID: \"85b72483-ec76-4bc2-a8ea-fdc16691d30e\") "
Sep 11 22:04:58.347449 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.347452    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703b93ef-d00e-4e9e-9e2e-6c407c7539d8-plugins-config-volume\") pod \"703b93ef-d00e-4e9e-9e2e-6c407c7539d8\" (UID: \"703b93ef-d00e-4e9e-9e2e-6c407c7539d8\") "
Sep 11 22:04:58.347683 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.347550    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b72483-ec76-4bc2-a8ea-fdc16691d30e-plugins-config-volume\") pod \"85b72483-ec76-4bc2-a8ea-fdc16691d30e\" (UID: \"85b72483-ec76-4bc2-a8ea-fdc16691d30e\") "
Sep 11 22:04:58.347683 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.347601    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7bfp\" (UniqueName: \"kubernetes.io/projected/703b93ef-d00e-4e9e-9e2e-6c407c7539d8-kube-api-access-w7bfp\") pod \"703b93ef-d00e-4e9e-9e2e-6c407c7539d8\" (UID: \"703b93ef-d00e-4e9e-9e2e-6c407c7539d8\") "
Sep 11 22:04:58.347928 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.347902    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/703b93ef-d00e-4e9e-9e2e-6c407c7539d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "703b93ef-d00e-4e9e-9e2e-6c407c7539d8" (UID: "703b93ef-d00e-4e9e-9e2e-6c407c7539d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:04:58.348021 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.347948    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b72483-ec76-4bc2-a8ea-fdc16691d30e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "85b72483-ec76-4bc2-a8ea-fdc16691d30e" (UID: "85b72483-ec76-4bc2-a8ea-fdc16691d30e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:04:58.349576 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.349551    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b72483-ec76-4bc2-a8ea-fdc16691d30e-kube-api-access-txb5v" (OuterVolumeSpecName: "kube-api-access-txb5v") pod "85b72483-ec76-4bc2-a8ea-fdc16691d30e" (UID: "85b72483-ec76-4bc2-a8ea-fdc16691d30e"). InnerVolumeSpecName "kube-api-access-txb5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:04:58.349699 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.349679    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/703b93ef-d00e-4e9e-9e2e-6c407c7539d8-kube-api-access-w7bfp" (OuterVolumeSpecName: "kube-api-access-w7bfp") pod "703b93ef-d00e-4e9e-9e2e-6c407c7539d8" (UID: "703b93ef-d00e-4e9e-9e2e-6c407c7539d8"). InnerVolumeSpecName "kube-api-access-w7bfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:04:58.448977 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.448879    2544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703b93ef-d00e-4e9e-9e2e-6c407c7539d8-plugins-config-volume\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:04:58.448977 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.448914    2544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b72483-ec76-4bc2-a8ea-fdc16691d30e-plugins-config-volume\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:04:58.448977 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.448927    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7bfp\" (UniqueName: \"kubernetes.io/projected/703b93ef-d00e-4e9e-9e2e-6c407c7539d8-kube-api-access-w7bfp\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:04:58.448977 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.448943    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txb5v\" (UniqueName: \"kubernetes.io/projected/85b72483-ec76-4bc2-a8ea-fdc16691d30e-kube-api-access-txb5v\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:04:58.635001 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.634959    2544 generic.go:356] "Generic (PLEG): container finished" podID="85b72483-ec76-4bc2-a8ea-fdc16691d30e" containerID="8c0d1cc2dfd765c347d178c6606741947ea46bcecd70476bc1d3d26450557e50" exitCode=0
Sep 11 22:04:58.635194 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.635010    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6"
Sep 11 22:04:58.635194 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.635040    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6" event={"ID":"85b72483-ec76-4bc2-a8ea-fdc16691d30e","Type":"ContainerDied","Data":"8c0d1cc2dfd765c347d178c6606741947ea46bcecd70476bc1d3d26450557e50"}
Sep 11 22:04:58.635194 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.635108    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6" event={"ID":"85b72483-ec76-4bc2-a8ea-fdc16691d30e","Type":"ContainerDied","Data":"26f79224dc71536a85d3e9f01cd20f83455ca95491f8071f1ba6b1d253ab0a41"}
Sep 11 22:04:58.635194 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.635143    2544 scope.go:117] "RemoveContainer" containerID="8c0d1cc2dfd765c347d178c6606741947ea46bcecd70476bc1d3d26450557e50"
Sep 11 22:04:58.636620 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.636590    2544 generic.go:356] "Generic (PLEG): container finished" podID="703b93ef-d00e-4e9e-9e2e-6c407c7539d8" containerID="9b6b01a9f6dd6e705ecf101f8830fcb88f9c0107bb4c3e25f26fcda64b3a5274" exitCode=0
Sep 11 22:04:58.636743 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.636627    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp"
Sep 11 22:04:58.636743 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.636675    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp" event={"ID":"703b93ef-d00e-4e9e-9e2e-6c407c7539d8","Type":"ContainerDied","Data":"9b6b01a9f6dd6e705ecf101f8830fcb88f9c0107bb4c3e25f26fcda64b3a5274"}
Sep 11 22:04:58.636743 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.636718    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp" event={"ID":"703b93ef-d00e-4e9e-9e2e-6c407c7539d8","Type":"ContainerDied","Data":"e75341598b945ff649d9d5a8f1cec58f3994c75f2b8ac0e2ac3b7158b24bdfc6"}
Sep 11 22:04:58.645845 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.645723    2544 scope.go:117] "RemoveContainer" containerID="8c0d1cc2dfd765c347d178c6606741947ea46bcecd70476bc1d3d26450557e50"
Sep 11 22:04:58.646319 ip-10-0-142-211 kubenswrapper[2544]: E0911 22:04:58.646287    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c0d1cc2dfd765c347d178c6606741947ea46bcecd70476bc1d3d26450557e50\": container with ID starting with 8c0d1cc2dfd765c347d178c6606741947ea46bcecd70476bc1d3d26450557e50 not found: ID does not exist" containerID="8c0d1cc2dfd765c347d178c6606741947ea46bcecd70476bc1d3d26450557e50"
Sep 11 22:04:58.646427 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.646332    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0d1cc2dfd765c347d178c6606741947ea46bcecd70476bc1d3d26450557e50"} err="failed to get container status \"8c0d1cc2dfd765c347d178c6606741947ea46bcecd70476bc1d3d26450557e50\": rpc error: code = NotFound desc = could not find container \"8c0d1cc2dfd765c347d178c6606741947ea46bcecd70476bc1d3d26450557e50\": container with ID starting with 8c0d1cc2dfd765c347d178c6606741947ea46bcecd70476bc1d3d26450557e50 not found: ID does not exist"
Sep 11 22:04:58.646427 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.646355    2544 scope.go:117] "RemoveContainer" containerID="9b6b01a9f6dd6e705ecf101f8830fcb88f9c0107bb4c3e25f26fcda64b3a5274"
Sep 11 22:04:58.658279 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.658255    2544 scope.go:117] "RemoveContainer" containerID="9b6b01a9f6dd6e705ecf101f8830fcb88f9c0107bb4c3e25f26fcda64b3a5274"
Sep 11 22:04:58.658639 ip-10-0-142-211 kubenswrapper[2544]: E0911 22:04:58.658615    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b6b01a9f6dd6e705ecf101f8830fcb88f9c0107bb4c3e25f26fcda64b3a5274\": container with ID starting with 9b6b01a9f6dd6e705ecf101f8830fcb88f9c0107bb4c3e25f26fcda64b3a5274 not found: ID does not exist" containerID="9b6b01a9f6dd6e705ecf101f8830fcb88f9c0107bb4c3e25f26fcda64b3a5274"
Sep 11 22:04:58.658703 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.658648    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b6b01a9f6dd6e705ecf101f8830fcb88f9c0107bb4c3e25f26fcda64b3a5274"} err="failed to get container status \"9b6b01a9f6dd6e705ecf101f8830fcb88f9c0107bb4c3e25f26fcda64b3a5274\": rpc error: code = NotFound desc = could not find container \"9b6b01a9f6dd6e705ecf101f8830fcb88f9c0107bb4c3e25f26fcda64b3a5274\": container with ID starting with 9b6b01a9f6dd6e705ecf101f8830fcb88f9c0107bb4c3e25f26fcda64b3a5274 not found: ID does not exist"
Sep 11 22:04:58.661058 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.661038    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6"]
Sep 11 22:04:58.663757 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.663732    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-077193-6d9544c74-lbct6"]
Sep 11 22:04:58.675636 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.675610    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp"]
Sep 11 22:04:58.679151 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:58.679124    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-077193-58cb899fb9-49zpp"]
Sep 11 22:04:59.592518 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:59.592416    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31206583-b73b-4dee-ba30-a2a822ca3733" path="/var/lib/kubelet/pods/31206583-b73b-4dee-ba30-a2a822ca3733/volumes"
Sep 11 22:04:59.592997 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:59.592963    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="703b93ef-d00e-4e9e-9e2e-6c407c7539d8" path="/var/lib/kubelet/pods/703b93ef-d00e-4e9e-9e2e-6c407c7539d8/volumes"
Sep 11 22:04:59.593392 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:04:59.593372    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85b72483-ec76-4bc2-a8ea-fdc16691d30e" path="/var/lib/kubelet/pods/85b72483-ec76-4bc2-a8ea-fdc16691d30e/volumes"
Sep 11 22:05:04.419898 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:04.419866    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"]
Sep 11 22:05:04.420400 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:04.420078    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq" podUID="7bc05684-948b-414b-9a71-ba8f47eaba93" containerName="istio-proxy" containerID="cri-o://79445d3b6e1e618a4b6312fccbbe4874c5053310380005282d50f180ed097104" gracePeriod=30
Sep 11 22:05:09.655527 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.655494    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:05:09.693363 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.693255    2544 generic.go:356] "Generic (PLEG): container finished" podID="7bc05684-948b-414b-9a71-ba8f47eaba93" containerID="79445d3b6e1e618a4b6312fccbbe4874c5053310380005282d50f180ed097104" exitCode=0
Sep 11 22:05:09.693363 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.693314    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"
Sep 11 22:05:09.693541 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.693360    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq" event={"ID":"7bc05684-948b-414b-9a71-ba8f47eaba93","Type":"ContainerDied","Data":"79445d3b6e1e618a4b6312fccbbe4874c5053310380005282d50f180ed097104"}
Sep 11 22:05:09.693541 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.693441    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq" event={"ID":"7bc05684-948b-414b-9a71-ba8f47eaba93","Type":"ContainerDied","Data":"f2b57ff7327bf1e2d91123956f525ccbf84562cff2a4e14201ed169231b676dd"}
Sep 11 22:05:09.693541 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.693475    2544 scope.go:117] "RemoveContainer" containerID="79445d3b6e1e618a4b6312fccbbe4874c5053310380005282d50f180ed097104"
Sep 11 22:05:09.701884 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.701862    2544 scope.go:117] "RemoveContainer" containerID="e10a3a133e13ffe61ec6ad3559f329a52c816bcebedda7acb7357627e30c7af9"
Sep 11 22:05:09.709608 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.709585    2544 scope.go:117] "RemoveContainer" containerID="79445d3b6e1e618a4b6312fccbbe4874c5053310380005282d50f180ed097104"
Sep 11 22:05:09.709870 ip-10-0-142-211 kubenswrapper[2544]: E0911 22:05:09.709855    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79445d3b6e1e618a4b6312fccbbe4874c5053310380005282d50f180ed097104\": container with ID starting with 79445d3b6e1e618a4b6312fccbbe4874c5053310380005282d50f180ed097104 not found: ID does not exist" containerID="79445d3b6e1e618a4b6312fccbbe4874c5053310380005282d50f180ed097104"
Sep 11 22:05:09.709944 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.709878    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79445d3b6e1e618a4b6312fccbbe4874c5053310380005282d50f180ed097104"} err="failed to get container status \"79445d3b6e1e618a4b6312fccbbe4874c5053310380005282d50f180ed097104\": rpc error: code = NotFound desc = could not find container \"79445d3b6e1e618a4b6312fccbbe4874c5053310380005282d50f180ed097104\": container with ID starting with 79445d3b6e1e618a4b6312fccbbe4874c5053310380005282d50f180ed097104 not found: ID does not exist"
Sep 11 22:05:09.709944 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.709893    2544 scope.go:117] "RemoveContainer" containerID="e10a3a133e13ffe61ec6ad3559f329a52c816bcebedda7acb7357627e30c7af9"
Sep 11 22:05:09.710124 ip-10-0-142-211 kubenswrapper[2544]: E0911 22:05:09.710109    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e10a3a133e13ffe61ec6ad3559f329a52c816bcebedda7acb7357627e30c7af9\": container with ID starting with e10a3a133e13ffe61ec6ad3559f329a52c816bcebedda7acb7357627e30c7af9 not found: ID does not exist" containerID="e10a3a133e13ffe61ec6ad3559f329a52c816bcebedda7acb7357627e30c7af9"
Sep 11 22:05:09.710168 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.710129    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10a3a133e13ffe61ec6ad3559f329a52c816bcebedda7acb7357627e30c7af9"} err="failed to get container status \"e10a3a133e13ffe61ec6ad3559f329a52c816bcebedda7acb7357627e30c7af9\": rpc error: code = NotFound desc = could not find container \"e10a3a133e13ffe61ec6ad3559f329a52c816bcebedda7acb7357627e30c7af9\": container with ID starting with e10a3a133e13ffe61ec6ad3559f329a52c816bcebedda7acb7357627e30c7af9 not found: ID does not exist"
Sep 11 22:05:09.741559 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.741519    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-workload-certs\") pod \"7bc05684-948b-414b-9a71-ba8f47eaba93\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") "
Sep 11 22:05:09.741751 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.741585    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl9zf\" (UniqueName: \"kubernetes.io/projected/7bc05684-948b-414b-9a71-ba8f47eaba93-kube-api-access-hl9zf\") pod \"7bc05684-948b-414b-9a71-ba8f47eaba93\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") "
Sep 11 22:05:09.741751 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.741608    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-token\") pod \"7bc05684-948b-414b-9a71-ba8f47eaba93\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") "
Sep 11 22:05:09.741751 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.741637    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-data\") pod \"7bc05684-948b-414b-9a71-ba8f47eaba93\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") "
Sep 11 22:05:09.741916 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.741825    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7bc05684-948b-414b-9a71-ba8f47eaba93" (UID: "7bc05684-948b-414b-9a71-ba8f47eaba93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:09.741916 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.741854    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-envoy\") pod \"7bc05684-948b-414b-9a71-ba8f47eaba93\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") "
Sep 11 22:05:09.741916 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.741887    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-workload-socket\") pod \"7bc05684-948b-414b-9a71-ba8f47eaba93\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") "
Sep 11 22:05:09.742084 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.741921    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bc05684-948b-414b-9a71-ba8f47eaba93-istiod-ca-cert\") pod \"7bc05684-948b-414b-9a71-ba8f47eaba93\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") "
Sep 11 22:05:09.742084 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.741948    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-credential-socket\") pod \"7bc05684-948b-414b-9a71-ba8f47eaba93\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") "
Sep 11 22:05:09.742084 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.741984    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-podinfo\") pod \"7bc05684-948b-414b-9a71-ba8f47eaba93\" (UID: \"7bc05684-948b-414b-9a71-ba8f47eaba93\") "
Sep 11 22:05:09.742323 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.742301    2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-workload-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:05:09.742382 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.741980    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-data" (OuterVolumeSpecName: "istio-data") pod "7bc05684-948b-414b-9a71-ba8f47eaba93" (UID: "7bc05684-948b-414b-9a71-ba8f47eaba93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:09.742382 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.742251    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7bc05684-948b-414b-9a71-ba8f47eaba93" (UID: "7bc05684-948b-414b-9a71-ba8f47eaba93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:09.742382 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.742307    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc05684-948b-414b-9a71-ba8f47eaba93-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7bc05684-948b-414b-9a71-ba8f47eaba93" (UID: "7bc05684-948b-414b-9a71-ba8f47eaba93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:05:09.742516 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.742434    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7bc05684-948b-414b-9a71-ba8f47eaba93" (UID: "7bc05684-948b-414b-9a71-ba8f47eaba93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:09.744012 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.743980    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7bc05684-948b-414b-9a71-ba8f47eaba93" (UID: "7bc05684-948b-414b-9a71-ba8f47eaba93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:05:09.744012 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.743988    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc05684-948b-414b-9a71-ba8f47eaba93-kube-api-access-hl9zf" (OuterVolumeSpecName: "kube-api-access-hl9zf") pod "7bc05684-948b-414b-9a71-ba8f47eaba93" (UID: "7bc05684-948b-414b-9a71-ba8f47eaba93"). InnerVolumeSpecName "kube-api-access-hl9zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:05:09.744292 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.744266    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-token" (OuterVolumeSpecName: "istio-token") pod "7bc05684-948b-414b-9a71-ba8f47eaba93" (UID: "7bc05684-948b-414b-9a71-ba8f47eaba93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:05:09.744385 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.744287    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7bc05684-948b-414b-9a71-ba8f47eaba93" (UID: "7bc05684-948b-414b-9a71-ba8f47eaba93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:09.843595 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.843555    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl9zf\" (UniqueName: \"kubernetes.io/projected/7bc05684-948b-414b-9a71-ba8f47eaba93-kube-api-access-hl9zf\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:05:09.843595 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.843588    2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-token\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:05:09.843595 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.843600    2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-data\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:05:09.843846 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.843611    2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-envoy\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:05:09.843846 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.843621    2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-workload-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:05:09.843846 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.843632    2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bc05684-948b-414b-9a71-ba8f47eaba93-istiod-ca-cert\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:05:09.843846 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.843643    2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc05684-948b-414b-9a71-ba8f47eaba93-credential-socket\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:05:09.843846 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:09.843652    2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bc05684-948b-414b-9a71-ba8f47eaba93-istio-podinfo\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:05:10.014535 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:10.014498    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"]
Sep 11 22:05:10.018333 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:10.018305    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-077193-openshift-default-54fc689874-gwrxq"]
Sep 11 22:05:11.592088 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:11.592050    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bc05684-948b-414b-9a71-ba8f47eaba93" path="/var/lib/kubelet/pods/7bc05684-948b-414b-9a71-ba8f47eaba93/volumes"
Sep 11 22:05:15.411145 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.411077    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7d7821-29819400-4wf67"]
Sep 11 22:05:15.413149 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.413123    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7d7821-29819400-4wf67"]
Sep 11 22:05:15.430581 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.430554    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl"]
Sep 11 22:05:15.430850 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.430821    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl" podUID="9b52ce0f-8e14-4114-9f65-60eb51ec7020" containerName="payload-processing" containerID="cri-o://33aadb6d57a336dbca7fd877f6e34095ffacc68e42782aa9daaffc20a101d4ef" gracePeriod=30
Sep 11 22:05:15.593722 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.593693    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcb2d38d-e320-4dae-8f1d-0100cc3624a9" path="/var/lib/kubelet/pods/dcb2d38d-e320-4dae-8f1d-0100cc3624a9/volumes"
Sep 11 22:05:15.672729 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.672669    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl"
Sep 11 22:05:15.691857 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.691826    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b52ce0f-8e14-4114-9f65-60eb51ec7020-plugins-config-volume\") pod \"9b52ce0f-8e14-4114-9f65-60eb51ec7020\" (UID: \"9b52ce0f-8e14-4114-9f65-60eb51ec7020\") "
Sep 11 22:05:15.692012 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.691898    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wvqb\" (UniqueName: \"kubernetes.io/projected/9b52ce0f-8e14-4114-9f65-60eb51ec7020-kube-api-access-9wvqb\") pod \"9b52ce0f-8e14-4114-9f65-60eb51ec7020\" (UID: \"9b52ce0f-8e14-4114-9f65-60eb51ec7020\") "
Sep 11 22:05:15.692323 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.692294    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b52ce0f-8e14-4114-9f65-60eb51ec7020-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9b52ce0f-8e14-4114-9f65-60eb51ec7020" (UID: "9b52ce0f-8e14-4114-9f65-60eb51ec7020"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:05:15.693993 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.693970    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b52ce0f-8e14-4114-9f65-60eb51ec7020-kube-api-access-9wvqb" (OuterVolumeSpecName: "kube-api-access-9wvqb") pod "9b52ce0f-8e14-4114-9f65-60eb51ec7020" (UID: "9b52ce0f-8e14-4114-9f65-60eb51ec7020"). InnerVolumeSpecName "kube-api-access-9wvqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:05:15.723919 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.723887    2544 generic.go:356] "Generic (PLEG): container finished" podID="9b52ce0f-8e14-4114-9f65-60eb51ec7020" containerID="33aadb6d57a336dbca7fd877f6e34095ffacc68e42782aa9daaffc20a101d4ef" exitCode=0
Sep 11 22:05:15.724005 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.723935    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl"
Sep 11 22:05:15.724005 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.723971    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl" event={"ID":"9b52ce0f-8e14-4114-9f65-60eb51ec7020","Type":"ContainerDied","Data":"33aadb6d57a336dbca7fd877f6e34095ffacc68e42782aa9daaffc20a101d4ef"}
Sep 11 22:05:15.724087 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.724031    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl" event={"ID":"9b52ce0f-8e14-4114-9f65-60eb51ec7020","Type":"ContainerDied","Data":"bc05840e57be4c0151da25f45c7f23bc75044d239b04a7684a442bed426d4370"}
Sep 11 22:05:15.724087 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.724056    2544 scope.go:117] "RemoveContainer" containerID="33aadb6d57a336dbca7fd877f6e34095ffacc68e42782aa9daaffc20a101d4ef"
Sep 11 22:05:15.732278 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.732256    2544 scope.go:117] "RemoveContainer" containerID="33aadb6d57a336dbca7fd877f6e34095ffacc68e42782aa9daaffc20a101d4ef"
Sep 11 22:05:15.732561 ip-10-0-142-211 kubenswrapper[2544]: E0911 22:05:15.732539    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33aadb6d57a336dbca7fd877f6e34095ffacc68e42782aa9daaffc20a101d4ef\": container with ID starting with 33aadb6d57a336dbca7fd877f6e34095ffacc68e42782aa9daaffc20a101d4ef not found: ID does not exist" containerID="33aadb6d57a336dbca7fd877f6e34095ffacc68e42782aa9daaffc20a101d4ef"
Sep 11 22:05:15.732651 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.732566    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33aadb6d57a336dbca7fd877f6e34095ffacc68e42782aa9daaffc20a101d4ef"} err="failed to get container status \"33aadb6d57a336dbca7fd877f6e34095ffacc68e42782aa9daaffc20a101d4ef\": rpc error: code = NotFound desc = could not find container \"33aadb6d57a336dbca7fd877f6e34095ffacc68e42782aa9daaffc20a101d4ef\": container with ID starting with 33aadb6d57a336dbca7fd877f6e34095ffacc68e42782aa9daaffc20a101d4ef not found: ID does not exist"
Sep 11 22:05:15.743512 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.743487    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl"]
Sep 11 22:05:15.745482 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.745460    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7d7821-5c5977d69-77fgl"]
Sep 11 22:05:15.792810 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.792786    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wvqb\" (UniqueName: \"kubernetes.io/projected/9b52ce0f-8e14-4114-9f65-60eb51ec7020-kube-api-access-9wvqb\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:05:15.792810 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:15.792811    2544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b52ce0f-8e14-4114-9f65-60eb51ec7020-plugins-config-volume\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:05:17.591746 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:17.591712    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b52ce0f-8e14-4114-9f65-60eb51ec7020" path="/var/lib/kubelet/pods/9b52ce0f-8e14-4114-9f65-60eb51ec7020/volumes"
Sep 11 22:05:32.859524 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:32.859490    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv"]
Sep 11 22:05:32.860108 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:32.859726    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv" podUID="f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6" containerName="payload-processing" containerID="cri-o://a3525855837024dd44c4b4871b86a833e5e5d471d62e9ce6626168e78d36d7c1" gracePeriod=30
Sep 11 22:05:32.864508 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:32.864477    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf"]
Sep 11 22:05:32.864743 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:32.864698    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf" podUID="01f0581f-2775-45fa-b005-33e7157f81bd" containerName="payload-pre-processing" containerID="cri-o://c4561ea10adf591dd6b2445123a59a5ebae27e3fd278e3af8f27611cea784708" gracePeriod=30
Sep 11 22:05:32.869277 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:32.869242    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf" podUID="01f0581f-2775-45fa-b005-33e7157f81bd" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.73:9004: connect: connection refused"
Sep 11 22:05:32.869968 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:32.869249    2544 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv" podUID="f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.74:9004: connect: connection refused"
Sep 11 22:05:33.119815 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.119787    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv"
Sep 11 22:05:33.122360 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.122340    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf"
Sep 11 22:05:33.245691 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.245660    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbnln\" (UniqueName: \"kubernetes.io/projected/f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6-kube-api-access-pbnln\") pod \"f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6\" (UID: \"f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6\") "
Sep 11 22:05:33.245859 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.245738    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f0581f-2775-45fa-b005-33e7157f81bd-plugins-config-volume\") pod \"01f0581f-2775-45fa-b005-33e7157f81bd\" (UID: \"01f0581f-2775-45fa-b005-33e7157f81bd\") "
Sep 11 22:05:33.245859 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.245784    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvpzc\" (UniqueName: \"kubernetes.io/projected/01f0581f-2775-45fa-b005-33e7157f81bd-kube-api-access-jvpzc\") pod \"01f0581f-2775-45fa-b005-33e7157f81bd\" (UID: \"01f0581f-2775-45fa-b005-33e7157f81bd\") "
Sep 11 22:05:33.245859 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.245809    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6-plugins-config-volume\") pod \"f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6\" (UID: \"f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6\") "
Sep 11 22:05:33.246198 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.246167    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f0581f-2775-45fa-b005-33e7157f81bd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01f0581f-2775-45fa-b005-33e7157f81bd" (UID: "01f0581f-2775-45fa-b005-33e7157f81bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:05:33.246345 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.246191    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6" (UID: "f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:05:33.247792 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.247769    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6-kube-api-access-pbnln" (OuterVolumeSpecName: "kube-api-access-pbnln") pod "f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6" (UID: "f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6"). InnerVolumeSpecName "kube-api-access-pbnln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:05:33.247859 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.247823    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f0581f-2775-45fa-b005-33e7157f81bd-kube-api-access-jvpzc" (OuterVolumeSpecName: "kube-api-access-jvpzc") pod "01f0581f-2775-45fa-b005-33e7157f81bd" (UID: "01f0581f-2775-45fa-b005-33e7157f81bd"). InnerVolumeSpecName "kube-api-access-jvpzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:05:33.346418 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.346381    2544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f0581f-2775-45fa-b005-33e7157f81bd-plugins-config-volume\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:05:33.346418 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.346411    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvpzc\" (UniqueName: \"kubernetes.io/projected/01f0581f-2775-45fa-b005-33e7157f81bd-kube-api-access-jvpzc\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:05:33.346418 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.346420    2544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6-plugins-config-volume\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:05:33.346648 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.346428    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbnln\" (UniqueName: \"kubernetes.io/projected/f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6-kube-api-access-pbnln\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:05:33.807135 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.807100    2544 generic.go:356] "Generic (PLEG): container finished" podID="f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6" containerID="a3525855837024dd44c4b4871b86a833e5e5d471d62e9ce6626168e78d36d7c1" exitCode=0
Sep 11 22:05:33.807341 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.807144    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv"
Sep 11 22:05:33.807341 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.807173    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv" event={"ID":"f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6","Type":"ContainerDied","Data":"a3525855837024dd44c4b4871b86a833e5e5d471d62e9ce6626168e78d36d7c1"}
Sep 11 22:05:33.807341 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.807246    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv" event={"ID":"f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6","Type":"ContainerDied","Data":"f3e363b93aba2d4c5d9a976cef26c980c74f29f818a3f3a064c84c70b69bf665"}
Sep 11 22:05:33.807341 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.807282    2544 scope.go:117] "RemoveContainer" containerID="a3525855837024dd44c4b4871b86a833e5e5d471d62e9ce6626168e78d36d7c1"
Sep 11 22:05:33.808748 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.808722    2544 generic.go:356] "Generic (PLEG): container finished" podID="01f0581f-2775-45fa-b005-33e7157f81bd" containerID="c4561ea10adf591dd6b2445123a59a5ebae27e3fd278e3af8f27611cea784708" exitCode=0
Sep 11 22:05:33.808876 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.808764    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf"
Sep 11 22:05:33.808876 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.808810    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf" event={"ID":"01f0581f-2775-45fa-b005-33e7157f81bd","Type":"ContainerDied","Data":"c4561ea10adf591dd6b2445123a59a5ebae27e3fd278e3af8f27611cea784708"}
Sep 11 22:05:33.808985 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.808876    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf" event={"ID":"01f0581f-2775-45fa-b005-33e7157f81bd","Type":"ContainerDied","Data":"81d16f67d51176f4aa8d4c449faf21e362300b845f763c5ae2e1547d5570a66b"}
Sep 11 22:05:33.815969 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.815957    2544 scope.go:117] "RemoveContainer" containerID="a3525855837024dd44c4b4871b86a833e5e5d471d62e9ce6626168e78d36d7c1"
Sep 11 22:05:33.816333 ip-10-0-142-211 kubenswrapper[2544]: E0911 22:05:33.816300    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3525855837024dd44c4b4871b86a833e5e5d471d62e9ce6626168e78d36d7c1\": container with ID starting with a3525855837024dd44c4b4871b86a833e5e5d471d62e9ce6626168e78d36d7c1 not found: ID does not exist" containerID="a3525855837024dd44c4b4871b86a833e5e5d471d62e9ce6626168e78d36d7c1"
Sep 11 22:05:33.816420 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.816340    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3525855837024dd44c4b4871b86a833e5e5d471d62e9ce6626168e78d36d7c1"} err="failed to get container status \"a3525855837024dd44c4b4871b86a833e5e5d471d62e9ce6626168e78d36d7c1\": rpc error: code = NotFound desc = could not find container \"a3525855837024dd44c4b4871b86a833e5e5d471d62e9ce6626168e78d36d7c1\": container with ID starting with a3525855837024dd44c4b4871b86a833e5e5d471d62e9ce6626168e78d36d7c1 not found: ID does not exist"
Sep 11 22:05:33.816420 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.816354    2544 scope.go:117] "RemoveContainer" containerID="c4561ea10adf591dd6b2445123a59a5ebae27e3fd278e3af8f27611cea784708"
Sep 11 22:05:33.824253 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.824183    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf"]
Sep 11 22:05:33.824324 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.824264    2544 scope.go:117] "RemoveContainer" containerID="c4561ea10adf591dd6b2445123a59a5ebae27e3fd278e3af8f27611cea784708"
Sep 11 22:05:33.824538 ip-10-0-142-211 kubenswrapper[2544]: E0911 22:05:33.824521    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4561ea10adf591dd6b2445123a59a5ebae27e3fd278e3af8f27611cea784708\": container with ID starting with c4561ea10adf591dd6b2445123a59a5ebae27e3fd278e3af8f27611cea784708 not found: ID does not exist" containerID="c4561ea10adf591dd6b2445123a59a5ebae27e3fd278e3af8f27611cea784708"
Sep 11 22:05:33.824602 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.824543    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4561ea10adf591dd6b2445123a59a5ebae27e3fd278e3af8f27611cea784708"} err="failed to get container status \"c4561ea10adf591dd6b2445123a59a5ebae27e3fd278e3af8f27611cea784708\": rpc error: code = NotFound desc = could not find container \"c4561ea10adf591dd6b2445123a59a5ebae27e3fd278e3af8f27611cea784708\": container with ID starting with c4561ea10adf591dd6b2445123a59a5ebae27e3fd278e3af8f27611cea784708 not found: ID does not exist"
Sep 11 22:05:33.827906 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.827886    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e0323-dcbd7bc4-76fcf"]
Sep 11 22:05:33.836951 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.836927    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv"]
Sep 11 22:05:33.842817 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:33.842799    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4e0323-745b4dd895-fl8rv"]
Sep 11 22:05:35.593414 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:35.593374    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01f0581f-2775-45fa-b005-33e7157f81bd" path="/var/lib/kubelet/pods/01f0581f-2775-45fa-b005-33e7157f81bd/volumes"
Sep 11 22:05:35.593923 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:05:35.593899    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6" path="/var/lib/kubelet/pods/f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6/volumes"
Sep 11 22:06:38.088805 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:06:38.088718    2544 scope.go:117] "RemoveContainer" containerID="d6426f321c9b3062717547235fe828afe099027dc99861cedefe4b50ab735a6b"
Sep 11 22:09:38.121805 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:09:38.121728    2544 scope.go:117] "RemoveContainer" containerID="ed04e48b3f0681d36f24f2b6125e5573a0e032d995382b91c0cc52dfbad03d4b"
Sep 11 22:10:14.569060 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569018    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"]
Sep 11 22:10:14.569700 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569532    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b72483-ec76-4bc2-a8ea-fdc16691d30e" containerName="payload-pre-processing"
Sep 11 22:10:14.569700 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569553    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b72483-ec76-4bc2-a8ea-fdc16691d30e" containerName="payload-pre-processing"
Sep 11 22:10:14.569700 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569569    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6" containerName="payload-processing"
Sep 11 22:10:14.569700 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569578    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6" containerName="payload-processing"
Sep 11 22:10:14.569700 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569591    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31206583-b73b-4dee-ba30-a2a822ca3733" containerName="revoke-keys"
Sep 11 22:10:14.569700 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569600    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="31206583-b73b-4dee-ba30-a2a822ca3733" containerName="revoke-keys"
Sep 11 22:10:14.569700 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569611    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b52ce0f-8e14-4114-9f65-60eb51ec7020" containerName="payload-processing"
Sep 11 22:10:14.569700 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569620    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b52ce0f-8e14-4114-9f65-60eb51ec7020" containerName="payload-processing"
Sep 11 22:10:14.569700 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569640    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc05684-948b-414b-9a71-ba8f47eaba93" containerName="istio-proxy"
Sep 11 22:10:14.569700 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569648    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc05684-948b-414b-9a71-ba8f47eaba93" containerName="istio-proxy"
Sep 11 22:10:14.569700 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569666    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01f0581f-2775-45fa-b005-33e7157f81bd" containerName="payload-pre-processing"
Sep 11 22:10:14.569700 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569674    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="01f0581f-2775-45fa-b005-33e7157f81bd" containerName="payload-pre-processing"
Sep 11 22:10:14.569700 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569687    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="703b93ef-d00e-4e9e-9e2e-6c407c7539d8" containerName="payload-processing"
Sep 11 22:10:14.569700 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569696    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="703b93ef-d00e-4e9e-9e2e-6c407c7539d8" containerName="payload-processing"
Sep 11 22:10:14.570344 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569791    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="85b72483-ec76-4bc2-a8ea-fdc16691d30e" containerName="payload-pre-processing"
Sep 11 22:10:14.570344 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569801    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4bbff87-a75a-4c7f-bb77-f8c612f0b8a6" containerName="payload-processing"
Sep 11 22:10:14.570344 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569811    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bc05684-948b-414b-9a71-ba8f47eaba93" containerName="istio-proxy"
Sep 11 22:10:14.570344 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569820    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="703b93ef-d00e-4e9e-9e2e-6c407c7539d8" containerName="payload-processing"
Sep 11 22:10:14.570344 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569830    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b52ce0f-8e14-4114-9f65-60eb51ec7020" containerName="payload-processing"
Sep 11 22:10:14.570344 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569839    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="31206583-b73b-4dee-ba30-a2a822ca3733" containerName="revoke-keys"
Sep 11 22:10:14.570344 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569848    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="01f0581f-2775-45fa-b005-33e7157f81bd" containerName="payload-pre-processing"
Sep 11 22:10:14.570344 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.569857    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bc05684-948b-414b-9a71-ba8f47eaba93" containerName="istio-proxy"
Sep 11 22:10:14.572736 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.572720    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.575099 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.575077    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e28fcd09-kserve-self-signed-certs\""
Sep 11 22:10:14.580876 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.580853    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"]
Sep 11 22:10:14.721850 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.721806    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-dshm\") pod \"e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") " pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.721850 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.721854    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e7b9e3-38f3-4054-a951-3efa96d00eca-tls-certs\") pod \"e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") " pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.722106 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.721946    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp4cr\" (UniqueName: \"kubernetes.io/projected/33e7b9e3-38f3-4054-a951-3efa96d00eca-kube-api-access-rp4cr\") pod \"e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") " pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.722106 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.722036    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-model-cache\") pod \"e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") " pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.722106 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.722081    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-tmp-dir\") pod \"e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") " pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.722246 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.722130    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-home\") pod \"e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") " pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.822798 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.822709    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp4cr\" (UniqueName: \"kubernetes.io/projected/33e7b9e3-38f3-4054-a951-3efa96d00eca-kube-api-access-rp4cr\") pod \"e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") " pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.822798 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.822757    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-model-cache\") pod \"e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") " pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.823030 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.822852    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-tmp-dir\") pod \"e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") " pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.823030 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.822874    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-home\") pod \"e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") " pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.823030 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.822889    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-dshm\") pod \"e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") " pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.823030 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.822913    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e7b9e3-38f3-4054-a951-3efa96d00eca-tls-certs\") pod \"e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") " pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.823295 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.823276    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-home\") pod \"e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") " pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.823377 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.823275    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-model-cache\") pod \"e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") " pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.823377 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.823314    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-tmp-dir\") pod \"e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") " pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.825148 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.825124    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-dshm\") pod \"e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") " pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.825379 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.825361    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e7b9e3-38f3-4054-a951-3efa96d00eca-tls-certs\") pod \"e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") " pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.830865 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.830843    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp4cr\" (UniqueName: \"kubernetes.io/projected/33e7b9e3-38f3-4054-a951-3efa96d00eca-kube-api-access-rp4cr\") pod \"e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") " pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:14.884281 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:14.884245    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:15.153681 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:15.153656    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"]
Sep 11 22:10:15.155779 ip-10-0-142-211 kubenswrapper[2544]: W0911 22:10:15.155751    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33e7b9e3_38f3_4054_a951_3efa96d00eca.slice/crio-71a37495a37d2ca247b3c47b221a2ceb946f64f4f4e06e255a8675723e91389e WatchSource:0}: Error finding container 71a37495a37d2ca247b3c47b221a2ceb946f64f4f4e06e255a8675723e91389e: Status 404 returned error can't find the container with id 71a37495a37d2ca247b3c47b221a2ceb946f64f4f4e06e255a8675723e91389e
Sep 11 22:10:16.053541 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:16.053499    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx" event={"ID":"33e7b9e3-38f3-4054-a951-3efa96d00eca","Type":"ContainerStarted","Data":"131e65b3f2a87e6874988d90c1e9d8b316ddee4ef6add1a7c66383550d2787c6"}
Sep 11 22:10:16.053931 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:16.053551    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx" event={"ID":"33e7b9e3-38f3-4054-a951-3efa96d00eca","Type":"ContainerStarted","Data":"71a37495a37d2ca247b3c47b221a2ceb946f64f4f4e06e255a8675723e91389e"}
Sep 11 22:10:16.071483 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:16.071427    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx" podStartSLOduration=2.071411315 podStartE2EDuration="2.071411315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:10:16.069102366 +0000 UTC m=+2379.038519187" watchObservedRunningTime="2026-09-11 22:10:16.071411315 +0000 UTC m=+2379.040828136"
Sep 11 22:10:20.655613 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.655573    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"]
Sep 11 22:10:20.656755 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.656729    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc05684-948b-414b-9a71-ba8f47eaba93" containerName="istio-proxy"
Sep 11 22:10:20.656890 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.656879    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc05684-948b-414b-9a71-ba8f47eaba93" containerName="istio-proxy"
Sep 11 22:10:20.660959 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.660938    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.663315 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.663297    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e28fcd09-kserve-self-signed-certs\""
Sep 11 22:10:20.666793 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.666773    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"]
Sep 11 22:10:20.780087 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.780048    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-dshm\") pod \"e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") " pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.780296 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.780111    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-model-cache\") pod \"e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") " pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.780296 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.780179    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-tmp-dir\") pod \"e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") " pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.780296 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.780254    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29pt4\" (UniqueName: \"kubernetes.io/projected/79660957-2e40-4102-bf4d-f79222761961-kube-api-access-29pt4\") pod \"e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") " pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.780456 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.780302    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-home\") pod \"e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") " pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.780456 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.780333    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79660957-2e40-4102-bf4d-f79222761961-tls-certs\") pod \"e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") " pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.881003 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.880965    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-model-cache\") pod \"e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") " pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.881003 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.881005    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-tmp-dir\") pod \"e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") " pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.881303 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.881105    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29pt4\" (UniqueName: \"kubernetes.io/projected/79660957-2e40-4102-bf4d-f79222761961-kube-api-access-29pt4\") pod \"e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") " pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.881303 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.881163    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-home\") pod \"e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") " pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.881303 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.881194    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79660957-2e40-4102-bf4d-f79222761961-tls-certs\") pod \"e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") " pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.881303 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.881253    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-dshm\") pod \"e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") " pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.881533 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.881428    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-model-cache\") pod \"e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") " pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.881533 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.881487    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-tmp-dir\") pod \"e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") " pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.881627 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.881546    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-home\") pod \"e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") " pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.883447 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.883421    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-dshm\") pod \"e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") " pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.883616 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.883598    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79660957-2e40-4102-bf4d-f79222761961-tls-certs\") pod \"e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") " pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.888917 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.888898    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29pt4\" (UniqueName: \"kubernetes.io/projected/79660957-2e40-4102-bf4d-f79222761961-kube-api-access-29pt4\") pod \"e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") " pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:20.971097 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:20.971007    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:21.245103 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:21.245058    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"]
Sep 11 22:10:21.247560 ip-10-0-142-211 kubenswrapper[2544]: W0911 22:10:21.247535    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79660957_2e40_4102_bf4d_f79222761961.slice/crio-5dcdcb186bd4d6ca3233e267a735d71bbc253acdaabed3f7adf105b5a003c9f0 WatchSource:0}: Error finding container 5dcdcb186bd4d6ca3233e267a735d71bbc253acdaabed3f7adf105b5a003c9f0: Status 404 returned error can't find the container with id 5dcdcb186bd4d6ca3233e267a735d71bbc253acdaabed3f7adf105b5a003c9f0
Sep 11 22:10:22.079458 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:22.079422    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t" event={"ID":"79660957-2e40-4102-bf4d-f79222761961","Type":"ContainerStarted","Data":"07d3231d1d01c950a9c67314cce626c9ef31df9be6a2324e60ce5ba317d29581"}
Sep 11 22:10:22.079458 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:22.079462    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t" event={"ID":"79660957-2e40-4102-bf4d-f79222761961","Type":"ContainerStarted","Data":"5dcdcb186bd4d6ca3233e267a735d71bbc253acdaabed3f7adf105b5a003c9f0"}
Sep 11 22:10:22.096431 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:22.096380    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t" podStartSLOduration=2.096363345 podStartE2EDuration="2.096363345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:10:22.095073482 +0000 UTC m=+2385.064490301" watchObservedRunningTime="2026-09-11 22:10:22.096363345 +0000 UTC m=+2385.065780166"
Sep 11 22:10:24.885268 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:24.885206    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:24.885685 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:24.885292    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:24.898146 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:24.898119    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:25.104058 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:25.104027    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:10:30.971573 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:30.971535    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:30.971961 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:30.971600    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:30.984079 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:30.984051    2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:31.131173 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:31.131139    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:10:38.138924 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:38.138891    2544 scope.go:117] "RemoveContainer" containerID="81a76fe1d0e418c94502d70aa3ac5b8d4bdf102514175db551175ae047325aa5"
Sep 11 22:10:58.190605 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:58.190570    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"]
Sep 11 22:10:58.968536 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:10:58.968505    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"]
Sep 11 22:11:13.209419 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.209307    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx" podUID="33e7b9e3-38f3-4054-a951-3efa96d00eca" containerName="main" containerID="cri-o://131e65b3f2a87e6874988d90c1e9d8b316ddee4ef6add1a7c66383550d2787c6" gracePeriod=45
Sep 11 22:11:13.459742 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.459688    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:11:13.548345 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.548308    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-model-cache\") pod \"33e7b9e3-38f3-4054-a951-3efa96d00eca\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") "
Sep 11 22:11:13.548345 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.548345    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-tmp-dir\") pod \"33e7b9e3-38f3-4054-a951-3efa96d00eca\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") "
Sep 11 22:11:13.548551 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.548360    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-home\") pod \"33e7b9e3-38f3-4054-a951-3efa96d00eca\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") "
Sep 11 22:11:13.548551 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.548387    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e7b9e3-38f3-4054-a951-3efa96d00eca-tls-certs\") pod \"33e7b9e3-38f3-4054-a951-3efa96d00eca\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") "
Sep 11 22:11:13.548551 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.548442    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-dshm\") pod \"33e7b9e3-38f3-4054-a951-3efa96d00eca\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") "
Sep 11 22:11:13.548551 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.548492    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp4cr\" (UniqueName: \"kubernetes.io/projected/33e7b9e3-38f3-4054-a951-3efa96d00eca-kube-api-access-rp4cr\") pod \"33e7b9e3-38f3-4054-a951-3efa96d00eca\" (UID: \"33e7b9e3-38f3-4054-a951-3efa96d00eca\") "
Sep 11 22:11:13.548713 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.548568    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33e7b9e3-38f3-4054-a951-3efa96d00eca" (UID: "33e7b9e3-38f3-4054-a951-3efa96d00eca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:11:13.548713 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.548600    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-model-cache" (OuterVolumeSpecName: "model-cache") pod "33e7b9e3-38f3-4054-a951-3efa96d00eca" (UID: "33e7b9e3-38f3-4054-a951-3efa96d00eca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:11:13.548713 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.548609    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-home" (OuterVolumeSpecName: "home") pod "33e7b9e3-38f3-4054-a951-3efa96d00eca" (UID: "33e7b9e3-38f3-4054-a951-3efa96d00eca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:11:13.548870 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.548773    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:11:13.548870 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.548795    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:11:13.548870 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.548807    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:11:13.550639 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.550612    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e7b9e3-38f3-4054-a951-3efa96d00eca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33e7b9e3-38f3-4054-a951-3efa96d00eca" (UID: "33e7b9e3-38f3-4054-a951-3efa96d00eca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:11:13.550731 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.550632    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-dshm" (OuterVolumeSpecName: "dshm") pod "33e7b9e3-38f3-4054-a951-3efa96d00eca" (UID: "33e7b9e3-38f3-4054-a951-3efa96d00eca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:11:13.550731 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.550643    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e7b9e3-38f3-4054-a951-3efa96d00eca-kube-api-access-rp4cr" (OuterVolumeSpecName: "kube-api-access-rp4cr") pod "33e7b9e3-38f3-4054-a951-3efa96d00eca" (UID: "33e7b9e3-38f3-4054-a951-3efa96d00eca"). InnerVolumeSpecName "kube-api-access-rp4cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:11:13.649161 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.649132    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp4cr\" (UniqueName: \"kubernetes.io/projected/33e7b9e3-38f3-4054-a951-3efa96d00eca-kube-api-access-rp4cr\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:11:13.649161 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.649161    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e7b9e3-38f3-4054-a951-3efa96d00eca-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:11:13.649349 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.649174    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e7b9e3-38f3-4054-a951-3efa96d00eca-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:11:13.987317 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:13.987270    2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t" podUID="79660957-2e40-4102-bf4d-f79222761961" containerName="main" containerID="cri-o://07d3231d1d01c950a9c67314cce626c9ef31df9be6a2324e60ce5ba317d29581" gracePeriod=45
Sep 11 22:11:14.262102 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.262078    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:11:14.301795 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.301759    2544 generic.go:356] "Generic (PLEG): container finished" podID="33e7b9e3-38f3-4054-a951-3efa96d00eca" containerID="131e65b3f2a87e6874988d90c1e9d8b316ddee4ef6add1a7c66383550d2787c6" exitCode=0
Sep 11 22:11:14.301963 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.301802    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"
Sep 11 22:11:14.301963 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.301840    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx" event={"ID":"33e7b9e3-38f3-4054-a951-3efa96d00eca","Type":"ContainerDied","Data":"131e65b3f2a87e6874988d90c1e9d8b316ddee4ef6add1a7c66383550d2787c6"}
Sep 11 22:11:14.301963 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.301886    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx" event={"ID":"33e7b9e3-38f3-4054-a951-3efa96d00eca","Type":"ContainerDied","Data":"71a37495a37d2ca247b3c47b221a2ceb946f64f4f4e06e255a8675723e91389e"}
Sep 11 22:11:14.301963 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.301915    2544 scope.go:117] "RemoveContainer" containerID="131e65b3f2a87e6874988d90c1e9d8b316ddee4ef6add1a7c66383550d2787c6"
Sep 11 22:11:14.303289 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.303264    2544 generic.go:356] "Generic (PLEG): container finished" podID="79660957-2e40-4102-bf4d-f79222761961" containerID="07d3231d1d01c950a9c67314cce626c9ef31df9be6a2324e60ce5ba317d29581" exitCode=0
Sep 11 22:11:14.303399 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.303302    2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"
Sep 11 22:11:14.303399 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.303349    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t" event={"ID":"79660957-2e40-4102-bf4d-f79222761961","Type":"ContainerDied","Data":"07d3231d1d01c950a9c67314cce626c9ef31df9be6a2324e60ce5ba317d29581"}
Sep 11 22:11:14.303399 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.303390    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t" event={"ID":"79660957-2e40-4102-bf4d-f79222761961","Type":"ContainerDied","Data":"5dcdcb186bd4d6ca3233e267a735d71bbc253acdaabed3f7adf105b5a003c9f0"}
Sep 11 22:11:14.310163 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.310143    2544 scope.go:117] "RemoveContainer" containerID="131e65b3f2a87e6874988d90c1e9d8b316ddee4ef6add1a7c66383550d2787c6"
Sep 11 22:11:14.310461 ip-10-0-142-211 kubenswrapper[2544]: E0911 22:11:14.310442    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"131e65b3f2a87e6874988d90c1e9d8b316ddee4ef6add1a7c66383550d2787c6\": container with ID starting with 131e65b3f2a87e6874988d90c1e9d8b316ddee4ef6add1a7c66383550d2787c6 not found: ID does not exist" containerID="131e65b3f2a87e6874988d90c1e9d8b316ddee4ef6add1a7c66383550d2787c6"
Sep 11 22:11:14.310554 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.310473    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131e65b3f2a87e6874988d90c1e9d8b316ddee4ef6add1a7c66383550d2787c6"} err="failed to get container status \"131e65b3f2a87e6874988d90c1e9d8b316ddee4ef6add1a7c66383550d2787c6\": rpc error: code = NotFound desc = could not find container \"131e65b3f2a87e6874988d90c1e9d8b316ddee4ef6add1a7c66383550d2787c6\": container with ID starting with 131e65b3f2a87e6874988d90c1e9d8b316ddee4ef6add1a7c66383550d2787c6 not found: ID does not exist"
Sep 11 22:11:14.310554 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.310496    2544 scope.go:117] "RemoveContainer" containerID="07d3231d1d01c950a9c67314cce626c9ef31df9be6a2324e60ce5ba317d29581"
Sep 11 22:11:14.318695 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.318667    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"]
Sep 11 22:11:14.318896 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.318873    2544 scope.go:117] "RemoveContainer" containerID="07d3231d1d01c950a9c67314cce626c9ef31df9be6a2324e60ce5ba317d29581"
Sep 11 22:11:14.319196 ip-10-0-142-211 kubenswrapper[2544]: E0911 22:11:14.319178    2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d3231d1d01c950a9c67314cce626c9ef31df9be6a2324e60ce5ba317d29581\": container with ID starting with 07d3231d1d01c950a9c67314cce626c9ef31df9be6a2324e60ce5ba317d29581 not found: ID does not exist" containerID="07d3231d1d01c950a9c67314cce626c9ef31df9be6a2324e60ce5ba317d29581"
Sep 11 22:11:14.319289 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.319205    2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d3231d1d01c950a9c67314cce626c9ef31df9be6a2324e60ce5ba317d29581"} err="failed to get container status \"07d3231d1d01c950a9c67314cce626c9ef31df9be6a2324e60ce5ba317d29581\": rpc error: code = NotFound desc = could not find container \"07d3231d1d01c950a9c67314cce626c9ef31df9be6a2324e60ce5ba317d29581\": container with ID starting with 07d3231d1d01c950a9c67314cce626c9ef31df9be6a2324e60ce5ba317d29581 not found: ID does not exist"
Sep 11 22:11:14.320325 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.320307    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e28fcd09-kserve-6fd7bb6495-lsgbx"]
Sep 11 22:11:14.355300 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.355274    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79660957-2e40-4102-bf4d-f79222761961-tls-certs\") pod \"79660957-2e40-4102-bf4d-f79222761961\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") "
Sep 11 22:11:14.355438 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.355315    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-home\") pod \"79660957-2e40-4102-bf4d-f79222761961\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") "
Sep 11 22:11:14.355438 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.355347    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-model-cache\") pod \"79660957-2e40-4102-bf4d-f79222761961\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") "
Sep 11 22:11:14.355438 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.355382    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29pt4\" (UniqueName: \"kubernetes.io/projected/79660957-2e40-4102-bf4d-f79222761961-kube-api-access-29pt4\") pod \"79660957-2e40-4102-bf4d-f79222761961\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") "
Sep 11 22:11:14.355438 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.355431    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-tmp-dir\") pod \"79660957-2e40-4102-bf4d-f79222761961\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") "
Sep 11 22:11:14.355682 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.355518    2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-dshm\") pod \"79660957-2e40-4102-bf4d-f79222761961\" (UID: \"79660957-2e40-4102-bf4d-f79222761961\") "
Sep 11 22:11:14.355682 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.355655    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-home" (OuterVolumeSpecName: "home") pod "79660957-2e40-4102-bf4d-f79222761961" (UID: "79660957-2e40-4102-bf4d-f79222761961"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:11:14.355789 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.355746    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "79660957-2e40-4102-bf4d-f79222761961" (UID: "79660957-2e40-4102-bf4d-f79222761961"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:11:14.355846 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.355807    2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-home\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:11:14.355931 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.355909    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-model-cache" (OuterVolumeSpecName: "model-cache") pod "79660957-2e40-4102-bf4d-f79222761961" (UID: "79660957-2e40-4102-bf4d-f79222761961"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:11:14.357480 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.357451    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79660957-2e40-4102-bf4d-f79222761961-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "79660957-2e40-4102-bf4d-f79222761961" (UID: "79660957-2e40-4102-bf4d-f79222761961"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:11:14.357596 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.357545    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-dshm" (OuterVolumeSpecName: "dshm") pod "79660957-2e40-4102-bf4d-f79222761961" (UID: "79660957-2e40-4102-bf4d-f79222761961"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:11:14.357853 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.357837    2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79660957-2e40-4102-bf4d-f79222761961-kube-api-access-29pt4" (OuterVolumeSpecName: "kube-api-access-29pt4") pod "79660957-2e40-4102-bf4d-f79222761961" (UID: "79660957-2e40-4102-bf4d-f79222761961"). InnerVolumeSpecName "kube-api-access-29pt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:11:14.456193 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.456157    2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29pt4\" (UniqueName: \"kubernetes.io/projected/79660957-2e40-4102-bf4d-f79222761961-kube-api-access-29pt4\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:11:14.456360 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.456208    2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-tmp-dir\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:11:14.456360 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.456237    2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-dshm\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:11:14.456360 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.456246    2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79660957-2e40-4102-bf4d-f79222761961-tls-certs\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:11:14.456360 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.456253    2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79660957-2e40-4102-bf4d-f79222761961-model-cache\") on node \"ip-10-0-142-211.ec2.internal\" DevicePath \"\""
Sep 11 22:11:14.623660 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.623628    2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"]
Sep 11 22:11:14.625504 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:14.625477    2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e28fcd09-kserve-58848569b7-cbv5t"]
Sep 11 22:11:15.591815 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:15.591785    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33e7b9e3-38f3-4054-a951-3efa96d00eca" path="/var/lib/kubelet/pods/33e7b9e3-38f3-4054-a951-3efa96d00eca/volumes"
Sep 11 22:11:15.592192 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:15.592184    2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79660957-2e40-4102-bf4d-f79222761961" path="/var/lib/kubelet/pods/79660957-2e40-4102-bf4d-f79222761961/volumes"
Sep 11 22:11:38.156713 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:38.156682    2544 scope.go:117] "RemoveContainer" containerID="adcc6db074953194ea7bf11d45d880518df28f4f6fe59d906f77b1fe1d9562e2"
Sep 11 22:11:49.158203 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:49.158170    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85c7cf647b-mj2pd_457c6b3b-046a-401b-9027-2a4355a5c253/authorino/0.log"
Sep 11 22:11:55.114143 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:55.114111    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-595996979b-k85d5_f22c40a2-230b-460a-874f-3f1d71d7a491/copy-manifests/0.log"
Sep 11 22:11:55.120342 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:55.120313    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-595996979b-k85d5_f22c40a2-230b-460a-874f-3f1d71d7a491/manager/0.log"
Sep 11 22:11:55.918524 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:55.918496    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819385-vxjr6_c9211812-da2c-4a91-9c92-a1ede1639ad5/cleanup/0.log"
Sep 11 22:11:56.917150 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:56.917107    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85c7cf647b-mj2pd_457c6b3b-046a-401b-9027-2a4355a5c253/authorino/0.log"
Sep 11 22:11:58.006118 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:58.006085    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs_c7429fa7-e597-47cd-9ef9-ceb95ded04dc/istio-proxy/0.log"
Sep 11 22:11:58.237474 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:58.237439    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57ddc9f5f6-4tg7c_0d4be6af-1f09-48dd-baac-d08a216c34e2/kube-auth-proxy/0.log"
Sep 11 22:11:58.837791 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:58.837761    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7448bd7fc8-p7xdg_162606cf-63cc-456f-89f4-5d222f5bb4f8/router/0.log"
Sep 11 22:11:59.199105 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:59.199024    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-dm7h7_c8c695c0-738f-4132-8cfb-25e62b7e8c2d/main/0.log"
Sep 11 22:11:59.432800 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:59.432770    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-8kbsp_7f0af874-2871-4f99-b7f9-0506f9bcf904/main/0.log"
Sep 11 22:11:59.550888 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:59.550859    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-7nqcx_62efe647-ce65-46b7-8937-3b7f534fa52b/main/0.log"
Sep 11 22:11:59.904267 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:11:59.904179    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-xltkj_a46b1d00-c254-4a5a-8adb-857e1aee0b25/main/0.log"
Sep 11 22:12:05.622588 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.622556    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wlxlj/must-gather-z8bmz"]
Sep 11 22:12:05.622978 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.622853    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79660957-2e40-4102-bf4d-f79222761961" containerName="main"
Sep 11 22:12:05.622978 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.622863    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="79660957-2e40-4102-bf4d-f79222761961" containerName="main"
Sep 11 22:12:05.622978 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.622878    2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e7b9e3-38f3-4054-a951-3efa96d00eca" containerName="main"
Sep 11 22:12:05.622978 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.622884    2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e7b9e3-38f3-4054-a951-3efa96d00eca" containerName="main"
Sep 11 22:12:05.622978 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.622936    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="79660957-2e40-4102-bf4d-f79222761961" containerName="main"
Sep 11 22:12:05.622978 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.622943    2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="33e7b9e3-38f3-4054-a951-3efa96d00eca" containerName="main"
Sep 11 22:12:05.626037 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.626020    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wlxlj/must-gather-z8bmz"
Sep 11 22:12:05.628281 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.628256    2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wlxlj\"/\"default-dockercfg-rq6g6\""
Sep 11 22:12:05.628397 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.628257    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wlxlj\"/\"kube-root-ca.crt\""
Sep 11 22:12:05.628500 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.628486    2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wlxlj\"/\"openshift-service-ca.crt\""
Sep 11 22:12:05.635615 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.635590    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wlxlj/must-gather-z8bmz"]
Sep 11 22:12:05.700060 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.700016    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnh8j\" (UniqueName: \"kubernetes.io/projected/b0a8718c-9617-49d8-a56d-f8cd253112a0-kube-api-access-cnh8j\") pod \"must-gather-z8bmz\" (UID: \"b0a8718c-9617-49d8-a56d-f8cd253112a0\") " pod="openshift-must-gather-wlxlj/must-gather-z8bmz"
Sep 11 22:12:05.700060 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.700061    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b0a8718c-9617-49d8-a56d-f8cd253112a0-must-gather-output\") pod \"must-gather-z8bmz\" (UID: \"b0a8718c-9617-49d8-a56d-f8cd253112a0\") " pod="openshift-must-gather-wlxlj/must-gather-z8bmz"
Sep 11 22:12:05.801598 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.801562    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnh8j\" (UniqueName: \"kubernetes.io/projected/b0a8718c-9617-49d8-a56d-f8cd253112a0-kube-api-access-cnh8j\") pod \"must-gather-z8bmz\" (UID: \"b0a8718c-9617-49d8-a56d-f8cd253112a0\") " pod="openshift-must-gather-wlxlj/must-gather-z8bmz"
Sep 11 22:12:05.801598 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.801603    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b0a8718c-9617-49d8-a56d-f8cd253112a0-must-gather-output\") pod \"must-gather-z8bmz\" (UID: \"b0a8718c-9617-49d8-a56d-f8cd253112a0\") " pod="openshift-must-gather-wlxlj/must-gather-z8bmz"
Sep 11 22:12:05.801962 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.801946    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b0a8718c-9617-49d8-a56d-f8cd253112a0-must-gather-output\") pod \"must-gather-z8bmz\" (UID: \"b0a8718c-9617-49d8-a56d-f8cd253112a0\") " pod="openshift-must-gather-wlxlj/must-gather-z8bmz"
Sep 11 22:12:05.810133 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.810106    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnh8j\" (UniqueName: \"kubernetes.io/projected/b0a8718c-9617-49d8-a56d-f8cd253112a0-kube-api-access-cnh8j\") pod \"must-gather-z8bmz\" (UID: \"b0a8718c-9617-49d8-a56d-f8cd253112a0\") " pod="openshift-must-gather-wlxlj/must-gather-z8bmz"
Sep 11 22:12:05.935903 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:05.935810    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wlxlj/must-gather-z8bmz"
Sep 11 22:12:06.208081 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:06.208056    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wlxlj/must-gather-z8bmz"]
Sep 11 22:12:06.210489 ip-10-0-142-211 kubenswrapper[2544]: W0911 22:12:06.210459    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0a8718c_9617_49d8_a56d_f8cd253112a0.slice/crio-b734e8612f42b7205635ced0140c55c7ed1c53634f779ee0342f0a1c7944c79b WatchSource:0}: Error finding container b734e8612f42b7205635ced0140c55c7ed1c53634f779ee0342f0a1c7944c79b: Status 404 returned error can't find the container with id b734e8612f42b7205635ced0140c55c7ed1c53634f779ee0342f0a1c7944c79b
Sep 11 22:12:06.212338 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:06.212320    2544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 22:12:06.532501 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:06.532453    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wlxlj/must-gather-z8bmz" event={"ID":"b0a8718c-9617-49d8-a56d-f8cd253112a0","Type":"ContainerStarted","Data":"b734e8612f42b7205635ced0140c55c7ed1c53634f779ee0342f0a1c7944c79b"}
Sep 11 22:12:07.542718 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:07.542675    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wlxlj/must-gather-z8bmz" event={"ID":"b0a8718c-9617-49d8-a56d-f8cd253112a0","Type":"ContainerStarted","Data":"f4edd49089f2012e655e1300e77294d93bfb88d4f1bb5dcbe9f35c337da64ae6"}
Sep 11 22:12:07.543231 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:07.542735    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wlxlj/must-gather-z8bmz" event={"ID":"b0a8718c-9617-49d8-a56d-f8cd253112a0","Type":"ContainerStarted","Data":"a70a3fc0f7ddcf4fc7e0657a2c9a8a00dc1aef3ffdf1747d5c38b595724c01d4"}
Sep 11 22:12:07.559353 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:07.559284    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wlxlj/must-gather-z8bmz" podStartSLOduration=1.8627410819999999 podStartE2EDuration="2.559259201s" totalImagesPullingTime="696.518119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:12:07.557452632 +0000 UTC m=+2490.526869449" watchObservedRunningTime="2026-09-11 22:12:07.559259201 +0000 UTC m=+2490.528676019"
Sep 11 22:12:08.459091 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:08.459055    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fjvb6_fef5eac9-cf6d-40b0-9bb2-374516793dc2/global-pull-secret-syncer/0.log"
Sep 11 22:12:08.597765 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:08.597733    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dkd6r_fec47591-6ba3-441d-8541-ee68a0470a2b/konnectivity-agent/0.log"
Sep 11 22:12:08.708347 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:08.708307    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-211.ec2.internal_400563a539e706696660090857d9ffd0/haproxy/0.log"
Sep 11 22:12:13.683255 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:13.682236    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85c7cf647b-mj2pd_457c6b3b-046a-401b-9027-2a4355a5c253/authorino/0.log"
Sep 11 22:12:15.575199 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:15.575150    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-4pmkk_d08eb046-1dfa-43cf-a851-0b722e18ba28/cluster-monitoring-operator/0.log"
Sep 11 22:12:15.789372 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:15.789344    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lttfq_c88f0aa7-fcd5-41b5-90ef-544e954dd7d6/node-exporter/0.log"
Sep 11 22:12:15.818452 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:15.818364    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lttfq_c88f0aa7-fcd5-41b5-90ef-544e954dd7d6/kube-rbac-proxy/0.log"
Sep 11 22:12:15.840065 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:15.839967    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lttfq_c88f0aa7-fcd5-41b5-90ef-544e954dd7d6/init-textfile/0.log"
Sep 11 22:12:17.903808 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:17.903768    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-4sjgx_545c8784-8a4a-433e-a576-4df8c2ec80e7/networking-console-plugin/0.log"
Sep 11 22:12:18.747167 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:18.747127    2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"]
Sep 11 22:12:18.755065 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:18.755034    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:18.758691 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:18.758658    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"]
Sep 11 22:12:18.842981 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:18.842943    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce41da56-68e3-44cf-9782-9d2726e83dab-sys\") pod \"perf-node-gather-daemonset-2kwb7\" (UID: \"ce41da56-68e3-44cf-9782-9d2726e83dab\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:18.843311 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:18.843288    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2dvh\" (UniqueName: \"kubernetes.io/projected/ce41da56-68e3-44cf-9782-9d2726e83dab-kube-api-access-h2dvh\") pod \"perf-node-gather-daemonset-2kwb7\" (UID: \"ce41da56-68e3-44cf-9782-9d2726e83dab\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:18.843473 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:18.843455    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce41da56-68e3-44cf-9782-9d2726e83dab-podres\") pod \"perf-node-gather-daemonset-2kwb7\" (UID: \"ce41da56-68e3-44cf-9782-9d2726e83dab\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:18.843683 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:18.843668    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce41da56-68e3-44cf-9782-9d2726e83dab-proc\") pod \"perf-node-gather-daemonset-2kwb7\" (UID: \"ce41da56-68e3-44cf-9782-9d2726e83dab\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:18.843873 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:18.843856    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce41da56-68e3-44cf-9782-9d2726e83dab-lib-modules\") pod \"perf-node-gather-daemonset-2kwb7\" (UID: \"ce41da56-68e3-44cf-9782-9d2726e83dab\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:18.944951 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:18.944910    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce41da56-68e3-44cf-9782-9d2726e83dab-proc\") pod \"perf-node-gather-daemonset-2kwb7\" (UID: \"ce41da56-68e3-44cf-9782-9d2726e83dab\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:18.945437 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:18.944994    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce41da56-68e3-44cf-9782-9d2726e83dab-lib-modules\") pod \"perf-node-gather-daemonset-2kwb7\" (UID: \"ce41da56-68e3-44cf-9782-9d2726e83dab\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:18.945437 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:18.945029    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce41da56-68e3-44cf-9782-9d2726e83dab-sys\") pod \"perf-node-gather-daemonset-2kwb7\" (UID: \"ce41da56-68e3-44cf-9782-9d2726e83dab\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:18.945437 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:18.945055    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce41da56-68e3-44cf-9782-9d2726e83dab-proc\") pod \"perf-node-gather-daemonset-2kwb7\" (UID: \"ce41da56-68e3-44cf-9782-9d2726e83dab\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:18.945437 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:18.945071    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dvh\" (UniqueName: \"kubernetes.io/projected/ce41da56-68e3-44cf-9782-9d2726e83dab-kube-api-access-h2dvh\") pod \"perf-node-gather-daemonset-2kwb7\" (UID: \"ce41da56-68e3-44cf-9782-9d2726e83dab\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:18.945437 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:18.945125    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce41da56-68e3-44cf-9782-9d2726e83dab-podres\") pod \"perf-node-gather-daemonset-2kwb7\" (UID: \"ce41da56-68e3-44cf-9782-9d2726e83dab\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:18.945437 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:18.945178    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce41da56-68e3-44cf-9782-9d2726e83dab-sys\") pod \"perf-node-gather-daemonset-2kwb7\" (UID: \"ce41da56-68e3-44cf-9782-9d2726e83dab\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:18.945437 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:18.945199    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce41da56-68e3-44cf-9782-9d2726e83dab-lib-modules\") pod \"perf-node-gather-daemonset-2kwb7\" (UID: \"ce41da56-68e3-44cf-9782-9d2726e83dab\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:18.945437 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:18.945364    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce41da56-68e3-44cf-9782-9d2726e83dab-podres\") pod \"perf-node-gather-daemonset-2kwb7\" (UID: \"ce41da56-68e3-44cf-9782-9d2726e83dab\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:18.953022 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:18.952996    2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2dvh\" (UniqueName: \"kubernetes.io/projected/ce41da56-68e3-44cf-9782-9d2726e83dab-kube-api-access-h2dvh\") pod \"perf-node-gather-daemonset-2kwb7\" (UID: \"ce41da56-68e3-44cf-9782-9d2726e83dab\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:19.070797 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:19.070725    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:19.354475 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:19.354441    2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"]
Sep 11 22:12:19.356058 ip-10-0-142-211 kubenswrapper[2544]: W0911 22:12:19.356026    2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podce41da56_68e3_44cf_9782_9d2726e83dab.slice/crio-0548a477697a08fbe65bab8d27ca65025c0e867a9e5bb0fd8b803eb6e5bdea2a WatchSource:0}: Error finding container 0548a477697a08fbe65bab8d27ca65025c0e867a9e5bb0fd8b803eb6e5bdea2a: Status 404 returned error can't find the container with id 0548a477697a08fbe65bab8d27ca65025c0e867a9e5bb0fd8b803eb6e5bdea2a
Sep 11 22:12:19.625488 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:19.625391    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7" event={"ID":"ce41da56-68e3-44cf-9782-9d2726e83dab","Type":"ContainerStarted","Data":"896875a8bc1a48e0ba1aea4383b1c56cca7d89f039026c22f6cd2de1addf22ef"}
Sep 11 22:12:19.625488 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:19.625461    2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7" event={"ID":"ce41da56-68e3-44cf-9782-9d2726e83dab","Type":"ContainerStarted","Data":"0548a477697a08fbe65bab8d27ca65025c0e867a9e5bb0fd8b803eb6e5bdea2a"}
Sep 11 22:12:19.626239 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:19.626199    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:19.642091 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:19.642033    2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7" podStartSLOduration=1.642014114 podStartE2EDuration="1.642014114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 22:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:12:19.640989499 +0000 UTC m=+2502.610406320" watchObservedRunningTime="2026-09-11 22:12:19.642014114 +0000 UTC m=+2502.611430933"
Sep 11 22:12:20.843627 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:20.843592    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pnzdj_77e62236-28b9-4261-85cb-b2aba118874b/dns/0.log"
Sep 11 22:12:20.861606 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:20.861578    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pnzdj_77e62236-28b9-4261-85cb-b2aba118874b/kube-rbac-proxy/0.log"
Sep 11 22:12:21.053833 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:21.053797    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6gf57_07c49e47-3378-4c6c-984a-802481fbf22c/dns-node-resolver/0.log"
Sep 11 22:12:21.674742 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:21.674715    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sh5wj_728ca319-dbd4-4ade-8a93-10191019f4df/node-ca/0.log"
Sep 11 22:12:22.594944 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:22.594899    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs_c7429fa7-e597-47cd-9ef9-ceb95ded04dc/istio-proxy/0.log"
Sep 11 22:12:23.440021 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:23.439983    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57ddc9f5f6-4tg7c_0d4be6af-1f09-48dd-baac-d08a216c34e2/kube-auth-proxy/0.log"
Sep 11 22:12:23.596394 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:23.596365    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7448bd7fc8-p7xdg_162606cf-63cc-456f-89f4-5d222f5bb4f8/router/0.log"
Sep 11 22:12:24.457497 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:24.457456    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dprjf_ae83570c-f2ee-4671-818e-f68d8c4cd4d0/serve-healthcheck-canary/0.log"
Sep 11 22:12:24.971629 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:24.971600    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2qg7x_92e79e38-c302-421c-886d-a0e68872587d/kube-rbac-proxy/0.log"
Sep 11 22:12:24.989971 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:24.989941    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2qg7x_92e79e38-c302-421c-886d-a0e68872587d/exporter/0.log"
Sep 11 22:12:25.008783 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:25.008753    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2qg7x_92e79e38-c302-421c-886d-a0e68872587d/extractor/0.log"
Sep 11 22:12:26.647203 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:26.647176    2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-2kwb7"
Sep 11 22:12:27.941279 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:27.941247    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-595996979b-k85d5_f22c40a2-230b-460a-874f-3f1d71d7a491/manager/0.log"
Sep 11 22:12:27.964005 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:27.963912    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-595996979b-k85d5_f22c40a2-230b-460a-874f-3f1d71d7a491/copy-manifests/0.log"
Sep 11 22:12:28.849247 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:28.849193    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs_c7429fa7-e597-47cd-9ef9-ceb95ded04dc/istio-proxy/0.log"
Sep 11 22:12:29.399937 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:29.399906    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-74f4c6fbc8-qhtdk_6188bae4-f1e3-4668-9b1f-1ddfab718dc8/manager/0.log"
Sep 11 22:12:29.682621 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:29.682500    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57ddc9f5f6-4tg7c_0d4be6af-1f09-48dd-baac-d08a216c34e2/kube-auth-proxy/0.log"
Sep 11 22:12:29.798814 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:29.798756    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7448bd7fc8-p7xdg_162606cf-63cc-456f-89f4-5d222f5bb4f8/router/0.log"
Sep 11 22:12:31.328337 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:31.328303    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-x9tcs_c7429fa7-e597-47cd-9ef9-ceb95ded04dc/istio-proxy/0.log"
Sep 11 22:12:32.203148 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:32.203116    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57ddc9f5f6-4tg7c_0d4be6af-1f09-48dd-baac-d08a216c34e2/kube-auth-proxy/0.log"
Sep 11 22:12:32.338201 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:32.338166    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7448bd7fc8-p7xdg_162606cf-63cc-456f-89f4-5d222f5bb4f8/router/0.log"
Sep 11 22:12:35.939472 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:35.939382    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rljhw_c44d4afa-8d65-4feb-83f9-7f6ed664511a/kube-multus-additional-cni-plugins/0.log"
Sep 11 22:12:35.962351 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:35.962321    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rljhw_c44d4afa-8d65-4feb-83f9-7f6ed664511a/egress-router-binary-copy/0.log"
Sep 11 22:12:35.980349 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:35.980322    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rljhw_c44d4afa-8d65-4feb-83f9-7f6ed664511a/cni-plugins/0.log"
Sep 11 22:12:35.998265 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:35.998235    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rljhw_c44d4afa-8d65-4feb-83f9-7f6ed664511a/bond-cni-plugin/0.log"
Sep 11 22:12:36.016628 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:36.016600    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rljhw_c44d4afa-8d65-4feb-83f9-7f6ed664511a/routeoverride-cni/0.log"
Sep 11 22:12:36.034867 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:36.034832    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rljhw_c44d4afa-8d65-4feb-83f9-7f6ed664511a/whereabouts-cni-bincopy/0.log"
Sep 11 22:12:36.052429 ip-10-0-142-211 kubenswrapper[2544]: I0911 22:12:36.052400    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rljhw_c44d4afa-8d65-4feb-83f9-7f6ed664511a/whereabouts-cni/0.log"